What is reinforcement learning?

As with many other machine learning, or more generally, AI problems, RL can also be intimidating if one starts directly from the full problem and the formal mathematical definitions, so let us start by loosely defining RL as a collection of both problems and representations, meaning that, we have both RL problems and RL methods to solve that class of problems. More formally, when we are working on a reinforcement learning problem, we are trying to map specific situations to an action or a set of actions, and each of those actions will have a consequence or a “reward” which can be either positive, neutral, or negative, in fact, this can simply be a real number.

For example, let’s say that we have a pet monkey called Marcel and that he has a set of toys that he loves to play with, and let’s say that we want to teach Marcel to pee in the toilet as opposed to on the floor, so to incentivize Marcel to choose the right action, we’ll give him a new toy every time he pees in the toilet (+1 toy) and we’ll remove a toy from his collection (-1 toy) every time he pees on the floor. In this case, hopefully, Marcel (we can call him the “agent”) will learn to select an “action” (pee on the floor vs pee in the toilet) whenever he finds himself in a given situation or “state” — when he feels the need to pee — in a way to maximize the number of toys, namely the rewards, by choosing the right actions at that state.

Now, I want to emphasize that while this example does a decent job describing the general idea of a reinforcement learning problem, there are many elements missing to fully describe the RL problem. A full reinforcement learning problem is usually defined under optimal control of Markov Decision Processes (MDPs), which rely on the Markov property that says that conditioned on the current state, future states are independent of past states.

If you’re familiarized with probability theory you might already know the Markov property or at least conditional probabilities, but we don’t necessarily need a thorough understanding of mathematical statistics to understand the idea behind the Markov property, so let’s go back to our previous example, which is Marcel’s toilet training. In this context, what the Markov property is telling us is that given that Marcel is in the state of “need to pee,” the state in which he’ll be next (peeing on the floor vs peeing in the toilet) is independent of whatever specific decisions — namely, the trajectory — he has made in the past (there are nuances to this but we’ll get to that in a minute.) Marcel could have peed on the floor twice and 4 times in the toilet or twice in the toilet and 0 times on the floor, in both cases, that leaves him with the need to pee and 2 new toys, and that defines his current state. In short, the Markov property tells us that given the present, the future doesn’t depend on the past.

If we stop to carefully evaluate this, we can relate this to another important property of reinforcement learning problems, which is the fact that they are (ideally) “closed-loop” and that this isn’t mutually exclusive with the Markov property.

Let’s use Marcel one more time to make this clearer, and let’s start by asking one question… Do you think Marcel will value a new toy once he has 100 toys in the same way as when he only had 2? Maybe not, so if Marcel finds himself in the state of “need to pee” and he has 100 toys already, he might not necessarily choose the same action he would have if he had only 2, so the actions Marcel chooses in the learning process can influence later inputs. But let’s be careful, this doesn’t mean that the specific past actions are conditioning future steps, because, notice that we are conditioning only in the current state, which is Marcel has to pee and he has 100 toys, it does not matter how he got to that state, he could have started his training with a 100 toys or he could have chosen the right action 100 times to get there, the only thing that matters is that he needs to pee and that he has 100 toys, so we are still making the future states independent from the past by conditioning on the present. Isn’t this beautiful?

It might feel like these are the same thing, or it might be very clear why the two previous statements are different, in any case, I’d encourage you to think about why they are different and perhaps try to visualize all of the possible “trajectories” that could lead Marcel to have 100 toys.

What are the pros and cons of reinforcement learning?

At this point, you might see that one of the biggest benefits of Reinforcement Learning is that we are not telling the “agent” or “learner” or Marcel, which actions he should take, rather we are setting up the problem in a way in which the agent will interact and explore an environment, select actions, and learn which actions to take to maximize the receiving rewards, and that means that the learner can actually improve over time and keep improving, and it will not be necessarily limited by a specific training dataset, which makes it very powerful to solve problems in non-stationary environments.

One of the drawbacks though is that training an agent can be very expensive, because of the same fact that the agent needs to actually interact with the environment to learn over time. Think about how expensive it would be to train an agent to drive a car. Although there are ways to work around this, with for example, simulators and other RL techniques, in general, training can be a huge limitation for many RL problems.

When should we consider applying reinforcement learning?

However, there are specific instances in which RL is a good fit to solve a problem, if the cost of “making mistakes” is very low, or simulators are relatively inexpensive, then RL is definitely a good candidate to solve that problem. In contrast, if making mistakes or access to high-quality simulators is difficult, perhaps a more simple approach can be used to find at least an acceptable solution.

What’s the difference between supervised learning and reinforcement learning?

That solution can be something like supervised learning, in which the “mapping” we do is more simple because we just try to map inputs to outputs (i.e. learn a function to map x to y). For example, when we try to teach Marcel what a toilet is, as opposed to how to use it, we could show him many pictures of different toilets, and eventually, he will know what a toilet is. Ok, perhaps we are stretching this example a little too far, but that is definitely the case for supervised learning, we can have a lot of features of the toilet, size, color, the sound it makes, etc. and assign a label to it (“toilet”) and use that to “recognize” toilets or at least recognize that something is not a toilet based on some of the object’s attributes. And that’s the main difference between supervised learning and reinforcement learning, in which we don’t tell the agent explicitly what actions to take, rather the agent learns which actions to take by interacting with the environment.

Types of reinforcement learning

Going back to reinforcement learning, there are a few variations that we haven’t discussed, and we won’t go into too many details about each, but before we do review some high-level details, let’s take a look at the anatomy of a reinforcement learning algorithm; we start by generating samples, meaning that we would “roll-out” and agent with a given policy (i.e. how to make decisions on which actions to take) and then we would see the rewards we get and with that data, we would estimate the total rewards of a given trajectory (set of decisions across time), then we use that information to improve our policy and then go back to generating samples and keep iterating. Below you can find a slide of Berkeley’s CS 285 course that has some details on the process.

There are two ways in which we can train the agent, “offline” and “online”. Online in this context means that we would be actively interacting with the environment and offline means that we are sampling from a fixed dataset.

What are the pros and cons of offline reinforcement learning?

A clear disadvantage of offline reinforcement learning is that because we are using a static dataset, the agent no longer has the ability to improve beyond the scope of the data provided by exploring different options and getting feedback from the environment. However, there is an upside, because we are using a fixed dataset, training can be more efficient and also less expensive compared to online reinforcement learning.

When should we consider applying offline reinforcement learning?

So we should consider applying offline reinforcement learning when the cost of simulators or online RL is not viable, and we should avoid using offline reinforcement learning when there’s the presence of severe distributional shifts, and that is because when we are using a fixed dataset, we can severely deviate from the “desired” trajectories over time.

Offline reinforcement learning in the real world

This is a lot of information to digest, but now that we understand the basic idea behind RL and also the definition of offline RL, let’s close the loop by providing an example of offline RL in the real world.

Let’s imagine that we are creating a system to support a decision-making process for situations in which the stakes are very high, for example, healthcare. We know that it would be both unethical and extremely dangerous to use online RL because the cost of selecting a bad action or making a mistake is extremely high. Therefore, in order to support the decision-making process (not replace it) we can use offline RL to train an agent using data that has information on different actions (possible interventions/treatments), observations that correspond to different symptoms, and the outcomes of each case. In settings such as this, offline RL might be the only viable option.

Hopefully, by now you have a better understanding of how reinforcement learning works, the basic structure of RL problems, the anatomy of RL algorithms, and why RL is important. If you’d like to dive deeper into the subject I would highly recommend following Berkeley’s CS 285 course, which is available online here.

References

1. CS 285 at UC Berkeley
2. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems