Objective

Learn exactly what it is that Tim Ferris believes makes a 4-hour workweek both a desirable and achievable goal.

Takeaways

There’s not a whole lot of plot to discuss, with the exception of the brief anecdotes that Tim indulges himself in, so we’ll just jump into the takeaways.

• The 4-hour workweek is largely a goal if you exclusively use your job as a means of generating cash to engage in other, “more enjoyable” activities
• Escaping the supposed drudgery of working a “typical” job means being efficient with time, cutting out distractions and being in an environment where getting work done is truly the determining factor. It also involves automating the laborious tasks we must complete but have little reason to focus on
• Escaping the confines of a typical job allow for people to explore the world they otherwise block out
• The goal in life is for continuous excitement rather than happiness, which involves constant development, rather than working towards a final ultimatum of retirement

While I can’t say I fully agree with these takeaways, I believe this list captures the essential points of Tim’s work. My main disagreement, in fact, arises from one of the central theses of his book, that is, that we are using our job solely as a means of raising capital. Instead, I greatly relish the chance to learn more about my fields of interest at work and can’t imagine a sustained activity that would give me as much enjoyment were it to replace this daily work. That being said, many of Tim’s points still apply in my case.

Point and case is the cutting out of unnecessary time wastes and distractions, such as messenging and email, and block off times for completing necessary errands. Rather than interrupting periods of great productivity, these activities can be siphoned off into periods relegated to such activities. This is definitely a point everyone can learn and implement, regardless of what you’re end goals are, seeing as it will make you more productive in whatever work or task it is you’re involved with. Could be developing, writing, recording music. Working without distractions and switch cost of multitasking greatly improves efficiency.

In addition, Tim’s point of sustained excitement is a surprisingly satisfying answer to being the primary goal of life. Sure, no simple sentence can truly capture this age-old discussion. But of the ones I’ve heard, this one resonates with me the most. This is largely because I can hardly picture myself retiring. I can barely sit at home for two days when there’s no work to be done without trying to find a new subject to learn or explore. His conception of life as a series of adventures hopping from one boat to the next with no final land to end on is definitely an appealing one.

Review

With all these points being said, I still think there’s plenty to glean from The 4-Hour Workweek. You’ll see how someone who’s goals of solely earning money from work can manifest most effectively. And even if, like me, this isn’t the case in your life, there’s still plenty you can learn from Tim’s streamlined lifestyle.

Few books give me as much pleasure discussing as Phil Knight’s Shoe Dog. While I strongly believe that different books hold varying appeal for different people, with some enjoying exploring the halls of Hogwarts and others learning the recounting of the troubles in Wall Street during the financial crisis, I strongly believe that everyone should at least give Shoe Dog a shot. Even if you’ve never read a biography before. Or even if you haven’t read a book in years. I can almost guarantee this will single-handedly remind you of what you’ve been missing — it sure reminded me. In any case, enough fan boying — let’s get to the actual discussion.

Objective

When reading in general, I strongly believe that having an objective in mind is necessary. As with anything you partake, there should be some rationale behind you investing time in doing it. That objective need not be related to work or business by any means — being entertained is a perfectly fine goal! But definitely let it be known to you that this is in fact your goal in beginning to read a book, since the way you approach this book will definitely be quite different than if you were doing it for something you want to influence your work. That being said, this is more explicitly elaborated in https://robertheaton.com/2018/06/25/how-to-read/, which I highly recommend go and check out.

My particular goal in reading Shoe Dog was to see first-hand how supposedly “traditional” businesses (that don’t have to do with the tech sector) are formed and the principles that Phil Knight followed to succeed.

Plot Summary

Shoe Dog recounts the story of Phil Knight, founder and eventual CEO of Nike, from its conception as a subsidiary of a Japanese shoe company, to its becoming a public US company. While most of us don’t think at all when we tie our shoes to go to school in the morning, this is what occupied the mind and work day of Knight for most of his life, and it is a truly fascinating experience to not only see this side of the business, but also the more messy, corporate mess he had to navigate to get Nike to the juggernaut it is today.

Takeaways

Three immediate takeaways become clear in reading the book, both about the business and Phil Knight himself. These were:

• Develop a business in which you have a personal stake, since it not only compels you to go to all the ends in the world to see it succeed, but also because you are more likely to know people from previous exposure in this familiar field than one that’s completely foreign to you
• The road taken to forming a successful company is often locally very circuitous, while remaining globally targeted
• Cultural understanding in a world business is crucial to success

Developing a business with a personal stake is almost always the first rule of thumb that business leaders will ascribe to. This was nowhere more clear in seeing how Phil Knight was able to navigate all the sh*t storms that would normally end a company: being screwed over by a deal from your corporate parents, getting investigated by the FBI, or losing a key player in your advertising campaign. While the details of how he evaded each of these circumstances largely evades my recall, what stuck was how his drive to see his designs brought to the market really compelled him to unearthly efforts to circumvent these pitfalls. Its this burning desire to see this idea come to fruition that brought Knight, along with many other visionaries, to push through all the points where, if they believed even one alternative could bring them as much satisfaction in its pursuit, they would have quit. Finding this singular drive is definitely a compelling point to admire in visionaries, especially as you see it unfold in Knight’s actions.

Much in this same line, Knight goes through so much more corporate drama than I could have imagined. And, it was really insightful to see that, while most leaders have a very composed appearance at the head of the company, they similarly go through all sorts of struggles in managing the mess that is an organization from its inside. Knight’s recount also showed his great indebtedness to luck and others he had met in his time in both business and outside alike. Seeing the direct ramifications of treating other business leaders, despite their clear supposed cultural differences, was enlightening, especially in the form of the contacts he retained in Japan.

Review

Honestly, the main thing that’s probably holding my ability to express how incredible Phil Knight’s Shoe Dog is is my writing ability. This truly is a must-read for anyone who can read. Go out and buy the book. Or buy the Kindle edition. Or buy the audiobook. Doesn’t matter how you read/listen to this book. You owe it to yourself to see how incredible the story of Nike truly is.

Like a relatively large portion of college students, I loved reading as a kid, only to let go of this hobby in entering high school. Swearing off reading for many of the years that followed, it was only after becoming friends with another student in my college class senior year of college that I truly rediscovered the joys of reading. Unlike my reading passions as a (younger) kid, however, I greatly relish reading non-fiction books, for the joy of getting to learn about a person, subject, or world that I would otherwise had little to no idea about. While I still very much love fantasy and sci-fi books (The Hitchhiker’s Guide to the Galaxy and Harry Potter will always have a special place in my mind), I’ve definitely started to slowly drift towards more non-fiction books.

With that in mind, I recently came across this journal article by Robert Heaton talking about learning “How to Read.” For anyone who hasn’t read this article, I highly recommend you take a moment and read it now (and then finish reading this article once you’re done): https://robertheaton.com/2018/06/25/how-to-read/. Not only did all the points in the beginning exposition resonate with me, but I had in fact become fully aware of these issues myself in the weeks before coming across this article. While I’ve known about this article for a couple months now, I’m only attempting to put his ideas in practice now.

And that brings me to the entire point of this particular blog post. I’m going to keep a running series of posts that go through the books that I’m reading to talk about my objectives in reading the book, a brief summary of its takeaways, my opinions on these takeaways, and a brief review. Hopefully this will help solidify my learnings from books while also inciting some interesting discussions about the matter.

For a general sense of the books I’m reading, feel free to check out my Goodreads page (https://www.goodreads.com/user/show/83094203-yash-patel) — highly recommend everyone get one! Anyway, this blog post was just intended as setup for the following blog posts that come out of my account, since I’ll be going through this exercise for the next week or so to get through the backlog of books that I’ve read over the last couple months!

Last time in our Keras/OpenAI tutorial, we discussed a very fundamental algorithm in reinforcement learning: the DQN. The Deep Q-Network is actually a fairly new advent that arrived on the seen only a couple years back, so it is quite incredible if you were able to understand and implement this algorithm having just gotten a start in the field. As with the original post, let’s take a quick moment to appreciate how incredible results we achieved are: in a continuous output space scenario and starting with absolutely no knowledge on what “winning” entails, we were able to explore our environment and “complete” the trials.

Put yourself in the situation of this simulation. This would essentially be like asking you to play a game, without a rulebook or specific endgoal, and demanding you to continue to play until you win (almost seems a bit cruel). And not only that: the possible result states you could reach with a series of actions is infinite (i.e. continuous observation space)! Yet, the DQN converges surprising quickly in tackling this seemingly impossible task by maintaining and slowly updating value internally to actions.

Even More Complex Environment

The step up from the previous MountainCar environment to the Pendulum is very similar to that from CartPole to MountainCar: we are expanding from a discrete environment to continuous. The Pendulum environment has an infinite input space, meaning that the number of actions you can take at any given time is unbounded. Why is DQN no longer applicable in this environment? Wasn’t our implementation of it completely independent of the structure of the environment actions?

While it was indepedent of what the actions were, the DQN was fundamentally premised on having a finite output space. After all, think about how we structured the code: the prediction looked to assign a score to each of the possible actions at each time step (given the current environment state) and simply taking the action that had the highest score. We had previously reduced the problem of reinforcement learning to effectively assigning scores to actions. But, how would this be possible if we have an infinite input space? We would need an infinitely large table to keep track of all the Q values!

So, how do we go about tackling this seemingly impossible task? After all, we’re being asked to do something even more insane than before: not only are we given a game without instructions to play and win, but this game has a controller with infinite buttons on it! Let’s see why it is that DQN is restricted to a finite number of actions.

The reason stems from how the model is structured: we have to be able to iterate at each time step to update how our position on a particular action has changed. That’s exactly why we were having the model predict the Q values rather than directly predicting what action to take. If we did the latter, we would have no idea how to update the model to take into account the prediction and what reward we received for future predictions. So, the fundamental issue stems from the fact that it seems like our model has to output a tabulated calculation of the rewards associated with all the possible actions. What if, instead, we broke this model apart? What if we had two separate models: one outputting the desired action (in the continuous space) and another taking in an action as input to produce the Q values from DQNs? That seems to solve our problems and is exactly the basis of the actor-critic model!

Actor-Critic Model Theory

As we went over in previous section, the entire Actor-Critic (AC) method is premised on having two interacting models. This theme of having multiple neural networks that interact is growing more and more relevant in both RL and supervised learning, i.e. GANs, AC, A3C, DDQN (dueling DQN), and so on. Getting familiar with these architectures may be somewhat intimidating the first time through but is certainly a worthwhile exercise: you’ll be able to understand and program some of the algorithms that are at the forefront of modern research in the field!

Getting back to the topic at hand, the AC model has two aptly named components: an actor and a critic. The former takes in the current environment state and determines the best action to take from there. It is essentially what would have seemed like the natural way to implement the DQN. The critic plays the “evaluation” role from the DQN by taking in the environment state and an action and returning a score that represents how apt the action is for the state.

Imagine this as a playground with a kid (the “actor”) and her parent (the “critic”). The kid is looking around, exploring all the possible options in this environment, such as sliding up a slide, swinging on a swing, and pulling grass from the ground. The parent will look at the kid, and either criticize or complement here based on what she did, taking the environment into account. The fact that the parent’s decision is environmentally-dependent is both important and intuitive: after all, if the child tried to swing on the swing, it would deserve far less praise than if she tried to do so on a slide!

Brief Aside: Chain Rule (Optional)

The main point of theory you need to understand is one that underpins a large part of modern-day machine learning: the chain rule. It would not be a tremendous overstatement to say that chain rule may be one of the most pivotal, even though somewhat simple, ideas to grasp to understand practical machine learning. In fact, you could probably get away with having little math background if you just intuitively understand what is conceptually convenyed by the chain rule. I’ll take a very quick aside to describe the chain rule, but if you feel quite comfortable with it, feel free to jump to the next section, where we actually see what the practical outline for developing the AC model looks like and how the chain rule fits into that plan.

Pictorially, this equation seems to make very intuitive sense: after all, just “cancel out the numerator/denominator.” There’s one major problem with this “intuitive explanation” though: the reasoning in this explanation is completely backwards! It is important to remember that math is just as much about developing intuitive notation as it is about understanding the concepts. And so, people developing this “fractional” notation because the chain rule behaves very similarly to simplifying fractional products. So, people who try to explain the concept just through the notation are skipping a key step: why is it that this notation is even applicable? As in, why do derivatives behave this way?

The underlying concept is actually not too much more difficult to grasp than this notation. Imagine we had a series of ropes that are tied together at some fixed points, similar to how springs in series would be attached. Let’s say you’re holding one end of this spring system and your goal is to shake the opposite end at some rate 10 ft/s. You could just shake your end at that speed and have it propagate to the other end. Or you could hook up some intermediary system that shakes the middle connection at some lower rate, i.e. at 5 ft/s. In that case, you’d only need to move your end at 2 ft/s, since whatever movement you’re making will be carried on from where you making the movement to the endpoint. This is because the physical connections force the movement on one end to be carried through to the end. Note: of course, as with any analogy, there are points of discrepancy here, but this was mostly for the purposes of visualization.

In a very similar way, if we have two systems where the output of one feeds into the input of the other, jiggling the parameters of the “feeding network,” will shake its output, which will propagate and be multiplied by any further changes through to the end of the pipeline.

AC Model Overview

Therefore, we have to develop an ActorCritic class that has some overlap with the DQN we previously implemented, but is more complex in its training. Because we’ll need some more advanced features, we’ll have to make use of the underlying library Keras rests upon: Tensorflow. Note: You can definitely implement this in Theano as well, but I haven’t worked with it in the past and so have not included its code. Feel free to submit expansions of this code to Theano if you choose to do so to me!

The model implementation will consist of four main parts, which directly parallel how we implemented the DQN agent:

• Model parameters/setup
• Training code
• Prediction code

AC Parameters

First off, just the imports we’ll be needing:

import gym
import numpy as np 
from keras.models import Sequential, Model
from keras.layers import Dense, Dropout, Input
from keras.layers.merge import Add, Multiply
from keras.optimizers import Adam
import keras.backend as K

import tensorflow as tf

import random
from collections import deque

The parameters are very similar to those in the DQN. After all, this actor-critic model has to do the same exact tasks as the DQN except in two separate modules. We also continue to use the “target network hack” that we discussed in the DQN post to ensure the network successfully converges. The only new parameter is referred to as “tau” and relates to a slight change in how the target network learning takes place in this case:

class ActorCritic:
    def __init__(self, env, sess):
        self.env  = env
        self.sess = sess

        self.learning_rate = 0.001
        self.epsilon = 1.0
        self.epsilon_decay = .995
        self.gamma = .95
        self.tau   = .125
        self.memory = deque(maxlen=2000)

The exact use of the tau parameter is explained more in the training section that follows but essentially plays the role of shifting from the prediction models to the target models gradually. Now, we reach the main points of interest: defining the models. As described, we have two separate models, each associated with its own target network.

We start with defining actor model. The purpose of the actor model is, given the current state of the environment, determine the best action to take. Once again, this task has numeric data that we are given, meaning there is no room or need to involve any more complex layers in the network than simply the Dense/fully-connected layers we’ve been using thus far. And so, the Actor model is quite simply a series of fully connected layers that maps from the environment observation to a point in the environment space:

def create_actor_model(self):
        state_input = Input(shape=self.env.observation_space.shape)
        h1 = Dense(24, activation='relu')(state_input)
        h2 = Dense(48, activation='relu')(h1)
        h3 = Dense(24, activation='relu')(h2)
        output = Dense(self.env.action_space.shape[0],  
            activation='relu')(h3)
        
        model = Model(input=state_input, output=output)
        adam  = Adam(lr=0.001)
        model.compile(loss="mse", optimizer=adam)
        return state_input, model

The main difference is that we return the a reference to the Input layer. The reason for this will be more clear by the end of this section, but briefly, it is for how we handle the training differently for the actor model.

The tricky part for the actor model comes in determining how to train it, and this is where the chain rule comes into play. But before we discuss that, let’s think about why it is any different than the standard critic/DQN network training. After all, aren’t we simply going to fit as in the DQN case, where we fit the model according to the current state and what the best action would be based on current and discounted future rewards? The problem lies in the question: if we’re able to do what we asked, then this would be a solved issue. The issue arises in how we determine what the “best action” to take would be, since the Q scores are now calculated separately in the critic network.

So, to overcome this, we choose an alternate approach. Rather than finding the “best option” and fitting on that, we essentially do hill climbing (gradient ascent). For those not familiar with the concept, hill climbing is a simple concept: from your local POV, determine the steepest direction of incline and move incrementally in that direction. In other words, hill climbing is attempting to reach a global max by simply doing the naive thing and following the directions of the local maxima. There are scenarios you could imagine where this would be hopelessly wrong, but more often than not, it works well in practical situations.

As a result, we want to use this approach to updating our actor model: we want to determine what change in parameters (in the actor model) would result in the largest increase in the Q value (predicted by the critic model). Since the output of the actor model is the action and the critic evaluates based on an environment state+action pair, we can see how the chain rule will play a role. We’ll want to see how changing the parameters of the actor will change the eventual Q, using the output of the actor network as our “middle link” (code below is all in the “__init__(self)” method):

        self.actor_state_input, self.actor_model = \
            self.create_actor_model()
        _, self.target_actor_model = self.create_actor_model()

        self.actor_critic_grad = tf.placeholder(tf.float32, 
            [None, self.env.action_space.shape[0]]) 
        
        actor_model_weights = self.actor_model.trainable_weights
        self.actor_grads = tf.gradients(self.actor_model.output, 
            actor_model_weights, -self.actor_critic_grad)
        grads = zip(self.actor_grads, actor_model_weights)
        self.optimize =  tf.train.AdamOptimizer(
            self.learning_rate).apply_gradients(grads)

We see that here we hold onto the gradient between the model weights and the output (action). We’ve also scaled it by the negation of self.actor_critic_grad (since we want to do gradient ascent in this case), which is held by a placeholder. For those unfamiliar with Tensorflow or learning for the first time, a placeholder plays the role of where you “input data” when you run the Tensorflow session. I won’t go into details about how it works, but the tensorflow.org tutorial goes through the material quite beautifully.

Moving on to the critic network, we are essentially faced with the opposite issue. That is, the network definition is slightly more complicated, but its training is relatively straightforward. The critic network is intended to take both the environment state and action as inputs and calculate a corresponding valuation. We do this by a series of fully-connected layers, with a layer in the middle that merges the two before combining into the final Q-value prediction:

def create_critic_model(self):
        state_input = Input(shape=self.env.observation_space.shape)
        state_h1 = Dense(24, activation='relu')(state_input)
        state_h2 = Dense(48)(state_h1)
        
        action_input = Input(shape=self.env.action_space.shape)
        action_h1    = Dense(48)(action_input)
        
        merged    = Add()([state_h2, action_h1])
        merged_h1 = Dense(24, activation='relu')(merged)
        output = Dense(1, activation='relu')(merged_h1)
        model  = Model(input=[state_input,action_input], 
            output=output)
        
        adam  = Adam(lr=0.001)
        model.compile(loss="mse", optimizer=adam)
        return state_input, action_input, model

The main points of note are the asymmetry in how we handle the inputs and what we’re returning. For the first point, we have one extra FC (fully-connected) layer on the environment state input as compared to the action. I did so because that is the recommended architecture for these AC networks, but it probably works equally (or marginally less) well with the FC layer slapped onto both inputs. As for the latter point (what we’re returning), we need to hold onto references of both the input state and action, since we need to use them in doing updates for the actor network:

        self.critic_state_input, self.critic_action_input, \
            self.critic_model = self.create_critic_model()
        _, _, self.target_critic_model = self.create_critic_model()

        self.critic_grads = tf.gradients(self.critic_model.output, 
            self.critic_action_input)
        
        # Initialize for later gradient calculations
        self.sess.run(tf.initialize_all_variables())

Here we set up the missing gradient to be calculated: the output Q with respect to the action weights. This is directly called in the training code, as we will now look into.

AC Training

The last main part of this code that is different from the DQN is the actual training. We do, however, make use of the same basic structure of pulling episodes from memory and learning from those. Since we have two training methods, we have separated the code into different training functions, cleanly calling them as:

def train(self):
        batch_size = 32
        if len(self.memory) < batch_size:
            return

        rewards = []
        samples = random.sample(self.memory, batch_size)
        self._train_critic(samples)
        self._train_actor(samples)

Now we define the two train methods. The, however, is very similar to that from the DQN: we are simply finding the discounted future reward and training on that. The only difference is that we’re training on the state/action pair and are using the target_critic_model to predict the future reward rather than the actor:

    def _train_critic(self, samples):
        for sample in samples:
            cur_state, action, reward, new_state, done = sample
            if not done:
                target_action = 
                    self.target_actor_model.predict(new_state)
                future_reward = self.target_critic_model.predict(
                    [new_state, target_action])[0][0]
                reward += self.gamma * future_reward
            self.critic_model.fit([cur_state, action], 
                reward, verbose=0)

As for the actor, we luckily did all the hard work before! We already set up how the gradients will work in the network and now simply have to call it with the actions and states we encounter:

def _train_actor(self, samples):
        for sample in samples:
            cur_state, action, reward, new_state, _ = sample
            predicted_action = self.actor_model.predict(cur_state)
            grads = self.sess.run(self.critic_grads, feed_dict={
                self.critic_state_input:  cur_state,
                self.critic_action_input: predicted_action
            })[0]

            self.sess.run(self.optimize, feed_dict={
                self.actor_state_input: cur_state,
                self.actor_critic_grad: grads
            })

As mentioned, we made use of the target model. And so, we have to update its weights at every time step. However, we only do so slowly. More concretely, we retain the value of the target model by a fraction self.tau and update it to be the corresponding model weight the remainder (1-self.tau) fraction. We do this for both the actor/critic, but only the actor is given below (you can see the critic in the full code at the bottom of the post):

def _update_actor_target(self):
        actor_model_weights  = self.actor_model.get_weights()
        actor_target_weights =self.target_critic_model.get_weights()
        
        for i in range(len(actor_target_weights)):
            actor_target_weights[i] = actor_model_weights[i]
        self.target_critic_model.set_weights(actor_target_weights

AC Prediction

This is identical to how we did it in the DQN, and so there’s not much to discuss on its implementation:

def act(self, cur_state):
  self.epsilon *= self.epsilon_decay
  if np.random.random() < self.epsilon:
   return self.env.action_space.sample()
  return self.actor_model.predict(cur_state)

Prediction Code

The prediction code is also very much the same as it was in previous reinforcement learning algorithms. That is, we just have to iterate through the trial and call predict, remember, and train on the agent:

def main():
    sess = tf.Session()
    K.set_session(sess)
    env = gym.make("Pendulum-v0")
    actor_critic = ActorCritic(env, sess)
    
    num_trials = 10000
    trial_len  = 500
    
    cur_state = env.reset()
    action = env.action_space.sample()
    while True:
        env.render()
        cur_state = cur_state.reshape((1, 
            env.observation_space.shape[0]))
        action = actor_critic.act(cur_state)
        action = action.reshape((1, env.action_space.shape[0]))
        
        new_state, reward, done, _ = env.step(action)
        new_state = new_state.reshape((1, 
            env.observation_space.shape[0]))
        
        actor_critic.remember(cur_state, action, reward, 
            new_state, done)
        actor_critic.train()
        
        cur_state = new_state

Complete Code

With that, here is the complete code used for training against the “Pendulum-v0” environment using AC (Actor-Critic)!

Boy, that was long: thanks for reading all the way through (or at least skimming)! Keep an eye out for the next Keras+OpenAI tutorial!

Comment and click that ❤️ below to show support!

Reinforcement Learning w/ Keras + OpenAI: Actor-Critic Models was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Last time in our Keras/OpenAI tutorial, we discussed a very basic example of applying deep learning to reinforcement learning contexts. This was an incredible showing in retrospect! If you looked at the training data, the random chance models would usually only be able to perform for 60 steps in median. And yet, by training on this seemingly very mediocre data, we were able to “beat” the environment (i.e. get >200 step performance). How is this possible?

We can get directly an intuitive feel for this. Let’s imagine the perfectly random series we used as our training data. It is extremely unlikely that any two series will have high overlap with one another, since these are generated completely randomly. However, there are key features that are common between successful trials, such as pushing the cart right when the pole is leaning right and vice versa. And so, by training our NN on all these trials data, we extract the shared patterns that contributed to them being successful and are able to smooth over the details that resulted in their independent failures.

That being said, the environment we consider this week is significantly more difficult than that from last week: the MountainCar.

More Complex Environments

Even though it seems we should be able to apply the same technique as that we applied last week, there is one key features here that makes doing so impossible: we can’t generate training data. Unlike the very simple Cartpole example, taking random movements often simply leads to the trial ending in us at the bottom of the hill. That is, we have several trials that are all identically -200 in the end. This is practically useless to use as training data. Imagine you were in a class where no matter what answers you put on your exam, you got a 0%! How are you going to learn from any of those experiences?

In line with that, we have to figure out a way to incrementally improve upon previous trials. For this, we use one of the most basic stepping stones for reinforcement learning: Q-learning!

DQN Theory Background

Q-learning (which doesn’t stand for anything, by the way) is centered around creating a “virtual table” that accounts for how much reward is assigned to each possible action given the current state of the environment. Let’s break that down one step at a time:

What do we mean by “virtual table?” Imagine that for each possible configuration of the input space, you have a table that assigns a score for each of the possible actions you can take. If this were magically possible, then it would be extremely easy for you to “beat” the environment: simply choose the action that has the highest score! Two points to note about this score. First, this score is conventionally referred to as the “Q-score,” which is where the name of the overall algorithm comes from. Second, as with any other score, these Q score have no meaning outside the context of their simulation. That is, they have no absolute significance, but that’s perfectly fine, since we solely need it to do comparisons.

Why then do we need virtual table for each input configuration? Why can’t we just have one table to rule them all? The reason is that it doesn’t make sense to do so: that would be the same as saying the best action to take while at the bottom of the valley is exactly that which you should take when you are perched on the highest point of the left incline.

Now, the main problem with what I described (maintaining a virtual table for each input configuration) is that this is impossible: we have a continuous (infinite) input space! We could get around this by discretizing the input space, but that seems like a pretty hacky solution to this problem that we’ll be encountering over and over in future situations. So, how do we get around this? By applying neural nets to the situation: that’s where the D in DQN comes from!

DQN Agent

So, we’ve now reduced the problem to finding a way to assign the different actions Q-scores given the current state. This is the answer to a very natural first question to answer when employing any NN: what are the inputs and outputs of our model? The extent of the math you need to understand for this model is the following equation (don’t worry, we’ll break it down):

Q, as mentioned, represents the value estimated by our model given the current state (s) and action taken (a). The goal, however, is to determine the overall value of a state. What do I mean by that? The overall value is both the immediate reward you will get and the expected rewards you will get in the future from being in that position. That is, we want to account for the fact that the value of a position often reflects not only its immediate gains but also the future gains it enables (damn, deep). In any case, we discount future rewards because, if I compare two situations in which I expect to get $100 one of the two being in the future, I would always take the present deal, since the position of the future one may change between when I made the deal and when I receive the money. The gamma factor reflects this depreciated value for the expected future returns on the state.

And that’s it: that’s all the math we’ll need for this! Time to actually move on to some code!

DQN Agent Implementation

The Deep Q Network revolves around continuous learning, meaning that we don’t simply accrue a bunch of trial/training data and feed it into the model. Instead, we create training data through the trials we run and feed this information into it directly after running the trial. If this all seems somewhat vague right now, don’t worry: time to see some code about this. The code largely revolves around defining a DQN class, where all the logic of the algorithm will actually be implemented, and where we expose a simple set of functions for the actual training.

DQN Hyperparameters

First off, we’re going to discuss some parameters of relevance for DQNs. Most of them are standard from most neural net implementations:

class DQN:
    def __init__(self, env):
        self.env     = env
        self.memory  = deque(maxlen=2000)
        
        self.gamma = 0.95
        self.epsilon = 1.0
        self.epsilon_min = 0.01
        self.epsilon_decay = 0.995
        self.learning_rate = 0.01

Let’s step through these one at a time. The first is simply the environment, which we supply for convenience when we need to reference the shapes in creating our model. The “memory” is a key component of DQNs: as mentioned previously, the trials are used to continuously train the model. However, rather than training on the trials as they come in, we add them to memory and train on a random sample of that memory. Why do this instead of just training on the last x trials as our “sample?” The reason is somewhat subtle. Imagine instead we were to just train on the most recent trials as our sample: in this case, our results would only learn on its most recent actions, which may not be directly relevant for future predictions. In this environment in particular, if we were moving down the right side of the slope, training on the most recent trials would entail training on the data where you were moving up the hill towards the right. But, this would not be at all relevant to determining what actions to take in the scenario you would soon be facing of scaling up the left hill. So, by taking a random sample, we don’t bias our training set, and instead ideally learn about scaling all environments we would encounter equally well.

So, we now discuss hyperparameters of the model: gamma, epsilon/epsilon decay, and the learning rate. The first is the future rewards depreciation factor (<1) discussed in the earlier equation, and the last is the standard learning rate parameter, so I won’t discuss that here. The second, however, is an interesting facet of RL that deserves a moment to discuss. In any sort of learning experience, we always have the choice between exploration vs. exploitation. This isn’t limited to computer science or academics: we do this on a day to day basis!

Consider the restaurants in your local neighborhood. When was the last time you went to a new one? Probably a long time ago. That corresponds to your shift from exploration to exploitation: rather than trying to find new and better opportunities, you settle with the best one you’ve found in your past experiences and maximize your utility from there. Contrast that to when you moved into your house: at that time, you had no idea what restaurants were good or not and so were enticed to explore your options. In other words, there’s a clear trend for learning: explore all your options when you’re unaware of them, and gradually shift over to exploiting once you’ve established opinions on some of them. In the same manner, we want our model to capture this natural model of learning, and epsilon plays that role.

Epsilon denotes the fraction of time we will dedicate to exploring. That is, a fraction self.epsilon of the trials, we will simply take a random action rather than the one we would predict to be the best in that scenario. As stated, we want to do this more often than not in the beginning, before we form stabilizing valuations on the matter, and so initialize epsilon to close to 1.0 at the beginning and decay it by some fraction <1 at every successive time step.

DQN Models

There was one key thing that was excluded in the initialization of the DQN above: the actual model used for predictions! As in our original Keras RL tutorial, we are directly given the input and output as numeric vectors. So, there’s no need to employ more complex layers in our network other than fully connected layers. Specifically, we define our model just as:

def create_model(self):
        model   = Sequential()
        state_shape  = self.env.observation_space.shape
        model.add(Dense(24, input_dim=state_shape[0], 
            activation="relu"))
        model.add(Dense(48, activation="relu"))
        model.add(Dense(24, activation="relu"))
        model.add(Dense(self.env.action_space.n))
        model.compile(loss="mean_squared_error",
            optimizer=Adam(lr=self.learning_rate))
        return model

And use this to define the model and target model (explained below):

def __init__(self, env):
        self.env     = env
        self.memory  = deque(maxlen=2000)
        
        self.gamma = 0.95
        self.epsilon = 1.0
        self.epsilon_min = 0.01
        self.epsilon_decay = 0.995
        self.learning_rate = 0.01
        self.tau = .05

        self.model = self.create_model()
        # "hack" implemented by DeepMind to improve convergence
        self.target_model = self.create_model()

The fact that there are two separate models, one for doing predictions and one for tracking “target values” is definitely counter-intuitive. To be explicit, the role of the model (self.model) is to do the actual predictions on what action to take, and the target model (self.target_model) tracks what action we want our model to take.

Why not just have a single model that does both? After all, if something is predicting the action to take, shouldn’t it be implicitly determine what model we want our model to take? This is actually one of those “weird tricks” in deep learning that DeepMind developed to get convergence in the DQN algorithm. If you use a single model, it can (and often does) converge in simple environments (such as the CartPole). But, the reason it doesn’t converge in these more complex environments is because of how we’re training the model: as mentioned previously, we’re training it “on the fly.”

As a result, we are doing training at each time step and, if we used a single network, would also be essentially changing the “goal” at each time step. Think of how confusing that would be! That would be like if a teacher told you to go finish pg. 6 in your textbook and, by the time you finished half of it, she changed it to pg. 9, and by the time you finished half of that, she told you to do pg. 21! This, therefore, causes a lack of convergence by a lack of clear direction in which to employ the optimizer, i.e. the gradients are changing too rapidly for stable convergence. So, to compensate, we have a network that changes more slowly that tracks our eventual goal and one that is trying to achieve those.

DQN Training

The training involves three main steps: remembering, learning, and reorienting goals. The first is basically just adding to the memory as we go through more trials:

def remember(self, state, action, reward, new_state, done):
        self.memory.append([state, action, reward, new_state, done])

There’s not much of note here other than that we have to store the done phase for how we later update the reward function. Moving on to the main body of our DQN, we have the train function. This is where we make use of our stored memory and actively learn from what we’ve seen in the past. We start by taking a sample from our entire memory storage. From there, we handle each sample different. As we saw in the equation before, we want to update the Q function as the sum of the current reward and expected future rewards (depreciated by gamma). In the case we are at the end of the trials, there are no such future rewards, so the entire value of this state is just the current reward we received. In a non-terminal state, however, we want to see what the maximum reward we would receive would be if we were able to take any possible action, from which we get:

def replay(self):
        batch_size = 32
        if len(self.memory) < batch_size: 
            return

        samples = random.sample(self.memory, batch_size)
        for sample in samples:
            state, action, reward, new_state, done = sample
            target = self.target_model.predict(state)
            if done:
                target[0][action] = reward
            else:
                Q_future = max(
                    self.target_model.predict(new_state)[0])
                target[0][action] = reward + Q_future * self.gamma
            self.model.fit(state, target, epochs=1, verbose=0)

And finally, we have to reorient our goals, where we simply copy over the weights from the main model into the target one. Unlike the main train method, however, this target update is called less frequently:

def target_train(self):
        weights = self.model.get_weights()
        target_weights = self.target_model.get_weights()
        for i in range(len(target_weights)):
            target_weights[i] = weights[i]
        self.target_model.set_weights(target_weights)

DQN Action

The final step is simply getting the DQN to actually perform the desired action, which alternates based on the given epsilon parameter between taking a random action and one predicated on past training, as follows:

def act(self, state):
        self.epsilon *= self.epsilon_decay
        self.epsilon = max(self.epsilon_min, self.epsilon)
        if np.random.random() < self.epsilon:
            return self.env.action_space.sample()
        return np.argmax(self.model.predict(state)[0])

Training Agent

Training the agent now follows naturally from the complex agent we developed. We have to instantiate it, feed it the experiences as we encounter them, train the agent, and update the target network:

def main():
    env     = gym.make("MountainCar-v0")
    gamma   = 0.9
    epsilon = .95

    trials  = 100
    trial_len = 500

    updateTargetNetwork = 1000
    dqn_agent = DQN(env=env)
    steps = []
    for trial in range(trials):
        cur_state = env.reset().reshape(1,2)
        for step in range(trial_len):
            action = dqn_agent.act(cur_state)
            env.render()
            new_state, reward, done, _ = env.step(action)

            reward = reward if not done else -20
            print(reward)
            new_state = new_state.reshape(1,2)
            dqn_agent.remember(cur_state, action, 
                reward, new_state, done)
            
            dqn_agent.replay()
            dqn_agent.target_train()

            cur_state = new_state
            if done:
                break
        if step >= 199:
            print("Failed to complete trial")
        else:
            print("Completed in {} trials".format(trial))
            break

Complete Code

With that, here is the complete code used for training against the “MountainCar-v0” environment using DQN!

Keep an eye out for the next Keras+OpenAI tutorial!

Comment and click that ❤️ below to show support!

Reinforcement Learning w/ Keras + OpenAI: DQNs was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Quick Background

Reinforcement learning (RL) is a general umbrella term for any algorithm that does not require explicit pairs of data and their corresponding desired labels as is the case in traditional supervised learning but requires some numeric indication of “how a sample is.” This quality of the “goodness” of a sample has no meaning in an absolute sense. You can imagine this generically as your score in a video game. If the screen displays a score of “218,” that presumably carries absolutely no meaning to you, the gamer, unless you are aware of how difficult or easy it is to earn a point and what score you start with. And that’s basically the extent of background we’ll be delving into: there will be more in-depth discussions of RL in the future, but the code we go through in this post is a very basic example of RL and so does not entail any further meddling in the subject theory.

Keras Notes

For anyone just getting started in AI/ML programming, welcome! The field has grown so much in the past years that it is quite overwhelming to jump in just now. But there’s still plenty of time to get involved and learn in this massive field! In line with that, Keras is the library I’ll primarily be using for my tutorials to come, including this one. Keras is essentially a wrapper library for Tensorflow and Theano. Its interface is quite similar to that exposed by tflearn but is slightly more generic in its applicability to Theano as the backend. Take note though! The dimensions in Theano are slightly different from those in Tensorflow. I would recommend, therefore, to adjust your Keras to use TF as its backend to avoid any frustrations with dimensions going forward (should be the default when you install, if you have TF installed already).

You could also just as easily do this with TF, but Keras gives us the nice flexibility when getting started of not having to keep track of dimensions through convolutions and all that crap. Anyway, enough words: time to move to the code!

Code

We’ll be exploring the most basic OpenAI environment here: the CartPole! As a final quick note, you can find the instructions to install OpenAI’s gym package here: https://gym.openai.com/docs. Just running “sudo pip install gym” should work on most platforms.

The CartPole environment has a very simple premise: balance the pole on the cart.

Data Collection

The first part of any machine learning problem is gathering the data, and this one is no different. Luckily, OpenAI’s gym environment provides a very straightforward way of gathering data: we can essentially just run through the simulation many times and take random steps every time. OpenAI Gym environments are structured around two main parts: an observation space and an action space. We observe the former from the environment and use that to determine how best to update it through an action, i.e. based on the current state of the pole (observation), determine whether to move the cart left or right (action).

As a result, we need to take an action that fits in the scope of the allowable actions of the action space, which is of size 2 in this case (left or right). We take the output space to be one-hot encoded, the reason being we want the neural net to eventually predict the probability of moving left vs. right given the current state of the environment. In this case, we could get away with just having the output be a single 1x1 float matrix (i.e. a scalar) and round it for our final result, but the one-hot encoding practice can be more widely applied.

So, to accumulate the actions and corresponding observations, a first thought may simply be:

for _ in range(10000):
    observation = env.reset()
    training_sampleX, training_sampleY = [], []
    for step in range(sim_steps):
        action = np.random.randint(0, 2)
        one_hot_action = np.zeros(2)
        one_hot_action[action] = 1
        training_sampleX.append(observation)
        training_sampleY.append(one_hot_action)
        
        observation, reward, done, _ = env.step(action)
        if done:
            break
    trainingX += training_sampleX
    trainingY += training_sampleY

However, if we were to train on this, the final predictor would likely do no better than random chance. After all, “garbage in, garbage out”: we would be doing no more than feeding the neural net a collection of both good and bad samples and expecting it to solely learn from the good. If we take a step back, however, this is completely implausible, since a single sample is indistinguishable from any other, even comparing those that come from good trials and those from poor trials.

So, instead, we’ll be only looking at the samples that result in trials with high scores. That is, we want to filter the samples to only allow those that eventually result in high scores in their trials. In this case, we arbitrarily chose 50 to be the “minimum cutoff” to be considered a “good trial,” and only select those samples:

def gather_data(env):
    min_score = 50
    sim_steps = 500
    trainingX, trainingY = [], []

    scores = []
    for _ in range(10000):
        observation = env.reset()
        score = 0
        training_sampleX, training_sampleY = [], []
        for step in range(sim_steps):
            action = np.random.randint(0, 2)
            one_hot_action = np.zeros(2)
            one_hot_action[action] = 1
            training_sampleX.append(observation)
            training_sampleY.append(one_hot_action)
            
            observation, reward, done, _ = env.step(action)
            score += reward
            if done:
                break
        if score > min_score:
            scores.append(score)
            trainingX += training_sampleX
            trainingY += training_sampleY

    trainingX, trainingY = np.array(trainingX), np.array(trainingY)
    print("Average: {}".format(np.mean(scores)))
    print("Median: {}".format(np.median(scores)))
    return trainingX, trainingY

Model Definition

Now that we have the data, we need to go about defining the model. Before doing any machine learning problem, it is always worthwhile stepping back to consider what it is we’re modelling, specifically what the expected inputs and desired results are. In our case, we’ll be receiving the current state of the environment (i.e. the “observations” from before) and wish to predict the probabilities of moving in each of the two directions. From this, we can easily figure out which of the two to take by taking the max arg.

The model we use here is a very simple one: several fully-connected layers (a.k.a. Dense layers in Keras). These are often the final layers used in deep CNNs (Convolution Neural Networks), since they are the ones that combine all the feature maps or input layers into the final scalar values. Fully-connected layers essentially make up the backbone of neural networks and are what allow them to effectively map high dimensional functions, ignoring all the modern enhancements with convolutions, LSTMs, Dropout, etc….

The only one of these enhancements that is relevant here is Dropout, since it helps ensure we do not overfit on the training data. So, we essentially sandwich a Dropout layer between each fully connected mapping to make sure that no one layer of the mapping becomes reliant on any small subset of connections that are specifically apparent in the training data.

Finally, we need to determine the loss function that we’ll train against. Since we encoded the output space as a one-hot 2D vector, the natural choice becomes categorical cross entropy, given that we wish to identify the result as either left ([1,0]) or right ([0,1]). I won’t go in-depth about what cross entropy entails, but it’s a very worthwhile function to understand, given its prevalence in these sorts of problems. From a high level, the cross entropy is, given two distributions (a true underlying distribution and our model thereof), a measure of how much information we need to convey something drawn from the true distribution by the model distribution.

Therefore, we define the model as:

from keras.models import Sequential
from keras.layers import Dense, Dropout

def create_model():
    model = Sequential()
    model.add(Dense(128, input_shape=(4,), activation="relu"))
    model.add(Dropout(0.6))
    
    model.add(Dense(256, activation="relu"))
    model.add(Dropout(0.6))
    
    model.add(Dense(512, activation="relu"))
    model.add(Dropout(0.6))
    
    model.add(Dense(256, activation="relu"))
    model.add(Dropout(0.6))
    
    model.add(Dense(128, activation="relu"))
    model.add(Dropout(0.6))
    model.add(Dense(2, activation="softmax"))
    
    model.compile(
        loss="categorical_crossentropy",
        optimizer="adam",
        metrics=["accuracy"])
    return model

A few more subtle technical points of the model: each of the layers of the model have ReLU activations to allow the model to train more rapidly than with saturating activation functions, such as tanh and sigmoid. The model will likely also train in these cases, but would take far longer to converge than if using ReLU activation.

Prediction

From there, we can simply get our training data, train the model, and iterate through several trials to see how well our model performs!

import gym
import numpy as np

from data import gather_data
from model import create_model

def predict():
    env = gym.make("CartPole-v0")
    trainingX, trainingY = gather_data(env)
    model = create_model()
    model.fit(trainingX, trainingY, epochs=5)
    
    scores = []
    num_trials = 50
    sim_steps = 500

    for trial in range(num_trials):
        observation = env.reset()
        score = 0
        for step in range(sim_steps):
            action = np.argmax(model.predict(
                observation.reshape(1,4)))
            observation, reward, done, _ = env.step(action)
            score += reward
            if done:
                break
        scores.append(score)
    
    print(np.mean(scores))

Full Code

With that step-by-step, here is the complete source code of the OpenAI Cartpole Keras implementation!

Keep an eye out for the next Keras+OpenAI tutorial!

Comment and click that ❤️ below to show support!