In this post, I am going to be implementing the Continuous Control with Deep Reinforcement Learning Paper.

The paper proposes an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces.

We will go straight into its implementation to avoid a massively large post. I suggest reading the paper in its entirety to gain better intuition.

First, we install the necessary packages and create some necessary directories:

!pip3 install torch opencv-python matplotlib numpy gym swig gym[box2d] box2d box2d-kengz
!mkdir ddpg plots # We need these directories.

The paper used Ornstein-Uhlenbeck as the noise class, a Replay Buffer, a Critic Network, an Actor Network, and an Agent. The noise class will be used in the Actor class to add extra exploration noise to action selection.

class OUActionNoise():
    def __init__(self, mu, sigma=0.15, theta=0.2, dt=1e-2, x0=None): # The paper suggests these default values for sigma, theta and dt.
        self.theta = theta
        self.mu = mu
        self.sigma = sigma
        self.dt = dt
        self.x0 = x0
        self.reset() # Resets the temporal correlation

    def __call__(self):
        # This is the equation suggested by the paper.
        x = self.x_prev + self.theta * (self.mu - self.x_prev) * self.dt + self.sigma * np.sqrt(self.dt) * np.random.normal(size=self.mu.shape)
        self.x_prev = x # Setting x_prev to the current value is how you set the temporal correlations.

        return x
        
    # There is no value for x_prev so we do that in the reset function.
    def reset(self):
        self.x_prev = self.x0 if self.x0 is not None else np.zeros_like(self.mu)

This is our Replay Buffer per the paper’s recommendation:

class ReplayBuffer():
    def __init__(self, max_size, input_shape, n_actions): # We are adding n_actions because we have to save the number of actions in tuples
        self.mem_size = max_size
        self.mem_cntr = 0 # Keeps a track of the position of the most recently saved memory
        self.state_memory = np.zeros((self.mem_size, *input_shape)) # Allocating memory slots with placeholder 0 values
        self.new_state_memory = np.zeros((self.mem_size, *input_shape))
        self.action_memory = np.zeros((self.mem_size, n_actions))
        self.reward_memory = np.zeros(self.mem_size) # Only gets mem_size as shape because its a scalar value

        # In RL, there is a concept of terminal state. When the episode is over, the agent enters the terminal state
        # from which it receives no future rewards so the value of that terminal state is 0.
        self.terminal_memory = np.zeros(self.mem_size, dtype=np.bool)

    def store_transition(self, state, action, reward, state_, done):
        index = self.mem_cntr % self.mem_size # Any memory counter less tham memory size just returns the memory counter
        self.state_memory[index] = state
        self.action_memory[index] = action # Actions are just arrays of arrays
        self.reward_memory[index] = reward
        self.new_state_memory[index] = state_
        self.terminal_memory[index] = done

        self.mem_cntr += 1 # Increment every time a new memory is stored

    def sample_buffer(self, batch_size):
        max_mem = min(self.mem_cntr, self.mem_size) # The reason why we don't just use mem_cntr is because mem_cntr goes larger than mem_size

        batch = np.random.choice(max_mem, batch_size) # Batch is just a random value between 0 to max_mem at the size of batch_size

        # Gathering the states
        states = self.state_memory[batch]
        actions = self.action_memory[batch]
        rewards = self.reward_memory[batch]
        states_ = self.new_state_memory[batch]
        dones = self.terminal_memory[batch]

        return states, actions, rewards, states_, dones

The following two classes are the Critic and Actor Networks:

class CriticNetwork(nn.Module):
    def __init__(self, beta, input_dims, fc1_dims, fc2_dims, n_actions, name, # beta is the learning rate
                 chkpt_dir='/content/ddpg'):
        super(CriticNetwork, self).__init__()
        self.input_dims = input_dims

        # Fully connected layer 1 and 2 dimensions
        self.fc1_dims = fc1_dims
        self.fc2_dims = fc2_dims
        self.n_actions = n_actions
        self.name = name
        self.checkpoint_dir = chkpt_dir
        self.checkpoint_file = os.path.join(self.checkpoint_dir, name+'_ddpg')

        #### Layers of the Neural Network ####
        self.fc1 = nn.Linear(*self.input_dims, self.fc1_dims)
        self.fc2 = nn.Linear(self.fc1_dims, self.fc2_dims)

        self.bn1 = nn.LayerNorm(self.fc1_dims) # To help with convergence
        self.bn2 = nn.LayerNorm(self.fc2_dims)

        self.action_value = nn.Linear(self.n_actions, self.fc2_dims)
        
        self.q = nn.Linear(self.fc2_dims, 1)

        f1 = 1./np.sqrt(self.fc1.weight.data.size()[0])
        self.fc1.weight.data.uniform_(-f1, f1)
        self.fc1.bias.data.uniform_(-f1, f1)

        f2 = 1./np.sqrt(self.fc2.weight.data.size()[0])
        self.fc2.weight.data.uniform_(-f2, f2)
        self.fc2.bias.data.uniform_(-f2, f2)

        f3 = 0.003
        self.q.weight.data.uniform_(-f3, f3)
        self.q.bias.data.uniform_(-f3, f3)

        f4 = 1./np.sqrt(self.action_value.weight.data.size()[0])
        self.action_value.weight.data.uniform_(-f4, f4)
        self.action_value.bias.data.uniform_(-f4, f4)

        self.optimizer = optim.Adam(self.parameters(), lr=beta, weight_decay=0.01)
        self.device = T.device('cuda:0' if T.cuda.is_available() else 'cuda:1')

        self.to(self.device)

    def forward(self, state, action):
        state_value = self.fc1(state)
        state_value = self.bn1(state_value)
        state_value = F.relu(state_value)
        state_value = self.fc2(state_value)
        state_value = self.bn2(state_value)
        action_value = self.action_value(action)
        state_action_value = F.relu(T.add(state_value, action_value))
        state_action_value = self.q(state_action_value)

        return state_action_value

    def save_checkpoint(self):
        print('... saving checkpoint ...')
        T.save(self.state_dict(), self.checkpoint_file) # Creates a state dictionary where the keys are the names of the 
                                                        # parameters and the vals are the parameters themselves

    def load_checkpoint(self):
        print('... loading checkpoint ...')
        self.load_state_dict(T.load(self.checkpoint_file))

    def save_best(self):
        print('... saving best checkpoint ...')
        checkpoint_file = os.path.join(self.checkpoint_dir, self.name+'_best')
        T.save(self.state_dict(), checkpoint_file)

class ActorNetwork(nn.Module):
    def __init__(self, alpha, input_dims, fc1_dims, fc2_dims, n_actions, 
                 name, # The name is to distinguish between the regular actor network and target actor network and also 
                       # to allow us to have multiple agents with different names.
                 chkpt_dir='/content/ddpg'):
      
        super(ActorNetwork, self).__init__()
        self.input_dims = input_dims
        self.fc1_dims = fc1_dims
        self.fc2_dims = fc2_dims
        self.n_actions = n_actions # to accomodate bounds where the action is greater than -1
        self.name = name
        self.checkpoint_dir = chkpt_dir
        self.checkpoint_file = os.path.join(self.checkpoint_dir, name+'_ddpg')

        self.fc1 = nn.Linear(*self.input_dims, self.fc1_dims)
        self.fc2 = nn.Linear(self.fc1_dims, self.fc2_dims)

        self.bn1 = nn.LayerNorm(self.fc1_dims)
        self.bn2 = nn.LayerNorm(self.fc2_dims)

        self.mu = nn.Linear(self.fc2_dims, self.n_actions)

        f2 = 1./np.sqrt(self.fc2.weight.data.size()[0])
        self.fc2.weight.data.uniform_(-f2, f2)
        self.fc2.bias.data.uniform_(-f2, f2)

        f1 = 1./np.sqrt(self.fc1.weight.data.size()[0])
        self.fc1.weight.data.uniform_(-f1, f1)
        self.fc1.bias.data.uniform_(-f1, f1)

        f3 = 0.003
        self.mu.weight.data.uniform_(-f3, f3)
        self.mu.bias.data.uniform_(-f3, f3)

        self.optimizer = optim.Adam(self.parameters(), lr=alpha)
        self.device = T.device('cuda:0' if T.cuda.is_available() else 'cuda:1')

        self.to(self.device)

    def forward(self, state):
        x = self.fc1(state)
        x = self.bn1(x)
        x = F.relu(x)
        x = self.fc2(x)
        x = self.bn2(x)
        x = F.relu(x)
        x = T.tanh(self.mu(x)) # Binds it between -1 and +1

        return x

    def save_checkpoint(self):
        print('... saving checkpoint ...')
        T.save(self.state_dict(), self.checkpoint_file)

    def load_checkpoint(self):
        print('... loading checkpoint ...')
        self.load_state_dict(T.load(self.checkpoint_file))

    def save_best(self):
        print('... saving best checkpoint ...')
        checkpoint_file = os.path.join(self.checkpoint_dir, self.name+'_best')
        T.save(self.state_dict(), checkpoint_file)

Finally, the agent:

class Agent():
    def __init__(self, alpha, beta, input_dims, tau, n_actions, gamma=0.99,
                 max_size=1000000, fc1_dims=400, fc2_dims=300, # Default values from the paper 
                 batch_size=64):
      
        self.gamma = gamma
        self.tau = tau
        self.batch_size = batch_size
        self.alpha = alpha
        self.beta = beta

        self.memory = ReplayBuffer(max_size, input_dims, n_actions)
        self.noise = OUActionNoise(mu=np.zeros(n_actions))
        self.actor = ActorNetwork(alpha, input_dims, fc1_dims, fc2_dims, n_actions=n_actions, name='actor')
        self.critic = CriticNetwork(beta, input_dims, fc1_dims, fc2_dims, n_actions=n_actions, name='critic')
        self.target_actor = ActorNetwork(alpha, input_dims, fc1_dims, fc2_dims, n_actions=n_actions, name='target_actor')
        self.target_critic = CriticNetwork(beta, input_dims, fc1_dims, fc2_dims, n_actions=n_actions, name='target_critic')

        self.update_network_parameters(tau=1) # The purpose of this is to solve the problem of a moving target.
                                              # In Q Learning, if you use one network to calculate both the action and the value of that action
                                              # then you're chasing a moving target because you're updating that estimate every turn. This could lead to
                                              # divergence due to using the same parameter. The solution to that is to use a target network that learns the
                                              # values of these states and action combinations. Then, the other network is what learns the policy.
                                              # You'd also have to periodically overwrite the target networks parameters with the evaluation network network parameters.

    def choose_action(self, observation):
        self.actor.eval() # The actor must go into an evaluation mode.
                          # This doesn't perform an evalutation step. It just tells PyTorch that we don't want to calculate statistics
                          # for the batch normalization. This is critical because without it, the agent will not learn. This step isn't necessary
                          # if we're not doing batch normalization, which we are (since its critical for convergence). This step also applies when
                          # using Dropout.

        state = T.tensor([observation], dtype=T.float).to(self.actor.device)
        mu = self.actor.forward(state).to(self.actor.device)
        mu_prime = mu + T.tensor(self.noise(), dtype=T.float).to(self.actor.device)
        self.actor.train()

        return mu_prime.cpu().detach().numpy()[0] # Convert the mu_prime to np because OpenAI gym does not accept tensors

    # This function acts as an interface for the ReplayBuffer class
    def remember(self, state, action, reward, state_, done):
        self.memory.store_transition(state, action, reward, state_, done)

    def save_models(self):
        self.actor.save_checkpoint()
        self.target_actor.save_checkpoint()
        self.critic.save_checkpoint()
        self.target_critic.save_checkpoint()

    def load_models(self):
        self.actor.load_checkpoint()
        self.target_actor.load_checkpoint()
        self.critic.load_checkpoint()
        self.target_critic.load_checkpoint()

    def learn(self):
        if self.memory.mem_cntr < self.batch_size: # We don't want the agent to learn if we haven't at least filled
            return                                 # up the batch size of our memory buffer.

        states, actions, rewards, states_, done = self.memory.sample_buffer(self.batch_size) # These come back as numpy arrays
                                                                                             # So we have to convert them all to tensors (below)

        states = T.tensor(states, dtype=T.float).to(self.actor.device)
        states_ = T.tensor(states_, dtype=T.float).to(self.actor.device)
        actions = T.tensor(actions, dtype=T.float).to(self.actor.device)
        rewards = T.tensor(rewards, dtype=T.float).to(self.actor.device)
        done = T.tensor(done).to(self.actor.device)

        # Calculating the target actions states like in Q Learning
        target_actions = self.target_actor.forward(states_)
        critic_value_ = self.target_critic.forward(states_, target_actions)
        critic_value = self.critic.forward(states, actions)

        critic_value_[done] = 0.0
        critic_value_ = critic_value_.view(-1)

        target = rewards + self.gamma*critic_value_
        target = target.view(self.batch_size, 1)

        self.critic.optimizer.zero_grad() # In Pytorch, whenever you calculate the loss function, you want to zero the gradients.
                                          # so the gradients from the previous states don't accumulate and interfere with the calculation.
                                          # it can slow stuff down.

        critic_loss = F.mse_loss(target, critic_value)
        critic_loss.backward()
        self.critic.optimizer.step()

        self.actor.optimizer.zero_grad() 
        actor_loss = -self.critic.forward(states, self.actor.forward(states))
        actor_loss = T.mean(actor_loss)
        actor_loss.backward()
        self.actor.optimizer.step()

        self.update_network_parameters()

    def update_network_parameters(self, tau=None): # Tau is a parameter that allows the update of the target network
                                                   # to gradually approach the evaluation networks and this is important for a nice slow convergence. You don't want to
                                                   # take too large steps in between updates so tau is a small number less than 1.
        if tau is None:
            tau = self.tau  # We are doing this because when we call the initializer in the beginning,
                            # we want all of the networks to start with the same weight.

        # Getting all of the names of the parameters from the networks.
        actor_params = self.actor.named_parameters()
        critic_params = self.critic.named_parameters()
        target_actor_params = self.target_actor.named_parameters()
        target_critic_params = self.target_critic.named_parameters()

        # Turning the names to dictionaries for easy iteration.
        critic_state_dict = dict(critic_params)
        actor_state_dict = dict(actor_params)
        target_critic_state_dict = dict(target_critic_params)
        target_actor_state_dict = dict(target_actor_params)

        for name in critic_state_dict:
            critic_state_dict[name] = tau*critic_state_dict[name].clone() + \
                                (1-tau)*target_critic_state_dict[name].clone()

        for name in actor_state_dict:
             actor_state_dict[name] = tau*actor_state_dict[name].clone() + \
                                 (1-tau)*target_actor_state_dict[name].clone()

        self.target_critic.load_state_dict(critic_state_dict)
        self.target_actor.load_state_dict(actor_state_dict)

The following function is just a custom plotting function. You can use Matplotlib’s pyplot if you wish.

def plot_learning_curve(x, scores, figure_file):
    running_avg = np.zeros(len(scores))

    for i in range(len(running_avg)):
        running_avg[i] = np.mean(scores[max(0, i-100):(i+1)])

    plt.plot(x, running_avg)
    plt.title('Running average of all scores')
    plt.savefig(figure_file)

We are basically training the agent against the Lundar Lander simulator by OpenAI. This is the training code:

env = gym.make('LunarLanderContinuous-v2')

agent = Agent(alpha=0.0001, 
              beta=0.001, 
              input_dims=env.observation_space.shape, 
              tau=0.001,
              batch_size=64, 
              fc1_dims=400, 
              fc2_dims=300, 
              n_actions=env.action_space.shape[0])

n_games = 1000
filename = 'LunarLander_alpha_' + str(agent.alpha) + '_beta_' + str(agent.beta) + '_' + str(n_games) + '_games'
          
figure_file = 'plots/' + filename + '.png'

best_score = env.reward_range[0]
score_history = []

for i in range(n_games):
    observation = env.reset()
    done = False
    score = 0
    agent.noise.reset()

    while not done:
        action = agent.choose_action(observation)
        observation_, reward, done, info = env.step(action)
        agent.remember(observation, action, reward, observation_, done)
        agent.learn()
        score += reward
        observation = observation_

    score_history.append(score)
    avg_score = np.mean(score_history)

    if avg_score > best_score:
        best_score = avg_score
        agent.save_models()

    print('episode ', i, 'score %.1f' % score,'average score %.1f' % avg_score)

x = [i+1 for i in range(n_games)]
plot_learning_curve(x, score_history, figure_file)

Here is the performance of the training…

episode  0 score -7.6 average score -7.6
episode  1 score -1256.8 average score -632.2
episode  2 score -521.4 average score -595.3
episode  3 score -571.8 average score -589.4
episode  4 score -541.8 average score -579.9
episode  5 score -1357.5 average score -709.5
episode  6 score -794.0 average score -721.5
episode  7 score -539.6 average score -698.8
episode  8 score -534.7 average score -680.6
episode  9 score -807.7 average score -693.3
episode  10 score -716.7 average score -695.4
episode  11 score -717.5 average score -697.2
episode  12 score -736.3 average score -700.3
...
episode  994 score 287.8 average score 127.1
... saving checkpoint ...
... saving checkpoint ...
... saving checkpoint ...
... saving checkpoint ...
episode  995 score 274.8 average score 127.2
... saving checkpoint ...
... saving checkpoint ...
... saving checkpoint ...
... saving checkpoint ...
episode  996 score 249.7 average score 127.3
... saving checkpoint ...
... saving checkpoint ...
... saving checkpoint ...
... saving checkpoint ...
episode  997 score 253.8 average score 127.5
... saving checkpoint ...
... saving checkpoint ...
... saving checkpoint ...
... saving checkpoint ...
episode  998 score 266.4 average score 127.6
... saving checkpoint ...
... saving checkpoint ...
... saving checkpoint ...
... saving checkpoint ...
episode  999 score 279.7 average score 127.7

(I’ve truncated the epochs because I trained the agent for 1000 steps).

Here is the plot:

By: Onur-Andros Ozbek

Montreal, Quebec

This is a project that I worked on during my internship. The goal is to basically detect logos that happen in gaming live streams from content creators, eSports events, etc. Large brands are heavily invested in these streams and they spend a lot of money on sponsorship so it only makes sense that they would want to know if the other side was holding their end of the deal.

We’ll get right into it. Let’s talk about the data (which was synthetic):

What I did was two things:

• Get a list of SVG files of logos of interest.
• Get many background images.
• Automatically annotate the logos with bounding boxes (since its an object detection problem)

The first part doesn’t require a lot of explanation. Let me explain how to go about the 2nd part.

Using the Twitch API and a Python FFmpeg wrapper, I took N number of Screenshots of M number of Twitch live streams. We need these frames to be able to paste our logos and create a dataset so this part is very important.

Once this was finished, I wrote a process that would take a random logo file, pass through an augmentation pipeline (rotation, cropping, perspective transformation, etc — based on percentages of what the augmentation is likely to look like in real life), paste on a random image and generate bounding box annotation file automatically in the YOLO format by saving the coordinates

(Unfortunately, I can’t share any of the code for the above process due to an NDA)

Once this was done, I went into training and inference on Colab. The following code is mostly provided by Roboflow with minor changes:

Configuring cuDNN on Colab for YOLOv4

# Change the number depending on what GPU is listed above, under NVIDIA-SMI > Name.
# Tesla K80: 30
# Tesla P100: 60
# Tesla T4: 75\
%env compute_capability=60

Installing Darknet for YOLOv4 on Colab

%cd /content/
%rm -rf darknet sample_data/

#we clone the fork of darknet maintained by roboflow
#small changes have been made to configure darknet for training
!git clone https://github.com/AlexeyAB/darknet.git

#install environment from the Makefile
%cd darknet/
# compute_30, sm_30 for Tesla K80
# compute_75, sm_75 for Tesla T4
# !sed -i 's/ARCH= -gencode arch=compute_60,code=sm_60/ARCH= -gencode arch=compute_30,code=sm_30/g' Makefile
 
#install environment from the Makefile
#note if you are on Colab Pro this works on a P100 GPU
#if you are on Colab free, you may need to change the Makefile for the K80 GPU
#this goes for any GPU, you need to change the Makefile to inform darknet which GPU you are running on.
!sed -i 's/OPENCV=0/OPENCV=1/g' Makefile
 
#for GPU accelerated training
!sed -i 's/GPU=0/GPU=1/g' Makefile
!sed -i 's/CUDNN=0/CUDNN=1/g' Makefile
!sed -i "s/ARCH= -gencode arch=compute_75,code=sm_75/ARCH= -gencode arch=compute_${compute_capability},code=sm_${compute_capability}/g" Makefile
!make

%cd /content/darknet

# YOLOv4weights
!wget https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.weights
!wget https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.conv.137

Set up Custom Dataset for YOLOv4

I have uploaded all the training and test images into my Google Drive. The cells below assemble the dataset to be compatible with the YOLO format.

# Mount Drive to Colab
from google.colab import drive
drive.mount('/content/drive')

%cd /content
!mkdir /content/darknet/prediction-images
!mkdir /content/darknet/data/obj

!unzip /content/drive/MyDrive/worlds-lec-dataset.zip -d /content/darknet/logorec/
!unzip /content/drive/MyDrive/youtube_images.zip -d /content/darknet/logorec/

!cp /content/drive/MyDrive/worlds-test-video.mp4 /content/darknet/
!cp /content/drive/MyDrive/lec-test-video.mp4 /content/darknet

%cat /content/darknet/logorec/Worlds-LEC-Dataset/classes.txt | cut -d" " -f2 | cut -d"." -f1 >> /content/darknet/data/obj.names

%cd /content/
!rm -rf /content/weights

# Keeping all imports in a single cell is a good idea so that, 
# if you need to update your imports and add more, you can just
# update one cell without having to re-run other code.

import os
from random import sample

trainshare = 0.8

#Set up training file directories for custom dataset
%cd /content/darknet/

#copy image and labels
%cp /content/darknet/logorec/Worlds-LEC-Dataset/*.png data/obj/
%cp /content/darknet/logorec/Worlds-LEC-Dataset/*.txt data/obj/

with open('data/obj.data', 'w') as out:
  out.write(f'classes = 13\n')
  out.write('train = data/train.txt\n')
  out.write('valid = data/valid.txt\n')
  out.write('names = data/obj.names\n')
  out.write('backup = backup/')
 
#write train file (just the image list)
files = [f for f in os.listdir('data/obj/') if f.endswith('png')]
train = sample(files, int(len(files) * trainshare))
validation = set(files) - set(train)

with open('data/train.txt', 'w') as out:
  for img in train:
    out.write('data/obj/' + img + '\n')
 
#write the valid file (just the image list)
with open('data/valid.txt', 'w') as out:
  for img in validation:
    out.write('data/obj/' + img + '\n')

Write Custom Training Config for YOLOv4

We need this config file for YOLOv4. I won’t post the entire thing here as it is way too long.

#we build config dynamically based on number of classes
#we build iteratively from base config files. This is the same file shape as cfg/yolo-obj.cfg
def file_len(fname):
  with open(fname) as f:
    for i, l in enumerate(f):
      pass
  return i + 1

num_classes = file_len('data/obj.names')
max_batches = num_classes*2000
steps1 = .8 * max_batches
steps2 = .9 * max_batches
steps_str = str(steps1)+','+str(steps2)
num_filters = (num_classes + 5) * 3
 
print("writing config for a custom YOLOv4 detector detecting number of classes: " + str(num_classes))
 
#Instructions from the darknet repo
#change line max_batches to (classes*2000 but not less than number of training images, and not less than 6000), f.e. max_batches=6000 if you train for 3 classes
#change line steps to 80% and 90% of max_batches, f.e. steps=4800,5400
if os.path.exists('./cfg/custom-yolov4-detector.cfg'): 
  os.remove('./cfg/custom-yolov4-detector.cfg')
 
#customize iPython writefile so we can write variables
from IPython.core.magic import register_line_cell_magic
 
@register_line_cell_magic
def writetemplate(line, cell):
    with open(line, 'w') as f:
        f.write(cell.format(**globals()))

Train Custom YOLOv4 Detector

Training is all about calling the training file.

!./darknet detector train data/obj.data cfg/custom-yolov4-detector.cfg yolov4.conv.137 -dont_show -map -clear

Running an inference on test videos and images with Saved YOLOv4 Weights

Once it is done, we want to test the model on a video file and save everything in our drive, including the weights.

%cd /content

!zip -r /content/drive/MyDrive/worlds-lec-dataset.zip /content/darknet/logorec/Worlds-LEC-Dataset
!zip -r /content/drive/MyDrive/youtube_images.zip /content/darknet/logorec/youtube_images

!cp /content/darknet/worlds-test-video.mp4 /content/drive/MyDrive/ 
!cp /content/darknet/lec-test-video.mp4 /content/drive/MyDrive/ 

#define utility function
def imShow(path):
  import cv2
  import matplotlib.pyplot as plt
  %matplotlib inline
 
  image = cv2.imread(path)
  height, width = image.shape[:2]
  resized_image = cv2.resize(image,(3*width, 3*height), interpolation = cv2.INTER_CUBIC)
 
  fig = plt.gcf()
  fig.set_size_inches(18, 10)
  plt.axis("off")
  plt.imshow(cv2.cvtColor(resized_image, cv2.COLOR_BGR2RGB))
  plt.show()

#check if weigths have saved yet
#backup houses the last weights for our detector
#(file yolo-obj_last.weights will be saved to the build\darknet\x64\backup\ for each 100 iterations)
#(file yolo-obj_xxxx.weights will be saved to the build\darknet\x64\backup\ for each 1000 iterations)
#After training is complete - get result yolo-obj_final.weights from path build\darknet\x64\bac
!ls /content/darknet/backup
#if it is empty you haven't trained for long enough yet, you need to train for at least 100 iterations

#coco.names is hardcoded somewhere in the detector
%cd /content/darknet/
%cp data/obj.names data/coco.names

The cell below runs inference on every single test image in the test folder.

import random, os, shutil

#/test has images that we can test our detector on
test_folder = "/content/darknet/logorec/youtube_images/"
test_images = [f for f in os.listdir(test_folder) if f.endswith('.png')]
img_path = ''
counter = 0

for i in range(0, len(test_images)):
  img_path = test_folder + test_images[i];
  !./darknet detect cfg/custom-yolov4-detector.cfg backup/custom-yolov4-detector_best.weights {img_path} -dont-show
  imShow('predictions.jpg')
  src_dir='/content/darknet/predictions.jpg'
  dst_dir='/content/darknet/prediction-images/prediction-'+str(counter)+".jpg"
  shutil.copy(src_dir,dst_dir)
  counter += 1

%cd /content/darknet/

Saving the weights, the config files, the charts, and the prediction images to the drive

!mkdir /content/darknet/training_results

# Creating the readme file
!touch /content/darknet/training_results/readme.txt
!echo "The training set was trained on YOLOv4's default weights with a distractor set" >> /content/darknet/training_results/readme.txt

# Moving the chart to the results directory
!cp /content/darknet/chart* /content/darknet/training_results

# Moving the config file to the results directory
!cp /content/darknet/cfg/custom-yolov4-detector.cfg /content/darknet/training_results

# Maving the obj.names and obj.data files to the results directory
!cp /content/darknet/data/obj.* /content/darknet/training_results

# Moving the weights to the results directory
!cp /content/darknet/backup/*.weights /content/darknet/training_results

# Moving the predicted images to the results directory  
!cp -r /content/darknet/prediction-images /content/darknet/training_results

# Zip the training result files and copy to the drive
!zip -r march13_results.zip /content/darknet/training_results
!cp march13_results.zip /content/drive/MyDrive/

!./darknet detector demo data/obj.data cfg/custom-yolov4-tiny-detector.cfg backup/custom-yolov4-tiny-detector_best.weights /content/darknet/worlds-test-video.mp4 -dont_show -thresh 0.25 -out_filename /content/RESULT_worlds-test.mp4
!cp /content/RESULT_LEC-test.mp4 /content/drive/MyDrive/

!./darknet detector demo data/obj.data cfg/custom-yolov4-tiny-detector.cfg backup/custom-yolov4-tiny-detector_best.weights /content/darknet/lec-test-video.mp4 -dont_show -thresh 0.25 -out_filename /content/RESULT_LEC-test.mp4
!cp /content/RESULT_worlds-test.mp4 /content/drive/MyDrive/

By:

Onur Andros Ozbek

Montreal, Quebec

Let's get right into it.

The shell commands below install the necessary packages. I had to downgrade PyTorch and Torchvision due to compatibility issues with Colab.

%reload_ext autoreload
%autoreload 2

!apt-get update
!apt install wget
!pip install pathlib
!pip install fastai
!pip install "torch==1.4" "torchvision==0.5.0"

The following are my imports. I’ve decided to use the FastAI library. Jeremy Howard had a great lecture on building classifiers. It is a simple library that allows developers to make use of PyTorch and it thought me a lot about multilabel classification.

Right underneath the imports, I also have the Label Smoothing class. Label smoothing has only been added to FastAI 2.0 beta at the time of this notebook so I had to manually include it.

import torch
import torch.nn as nn
from fastai import *
from fastai.vision import *
from fastai.callbacks.hooks import *
from pathlib import Path
import os, shutil

from torch.distributions.beta import Beta

def lin_comb(a, b, frac_a): return (frac_a * a) + (1 - frac_a) * b

def unsqueeze(input, dims):
    for dim in listify(dims): input = torch.unsqueeze(input, dim)
    return input

def reduce_loss(loss, reduction='mean'):
    return loss.mean() if reduction=='mean' else loss.sum() if reduction=='sum' else loss

class LabelSmoothingCrossEntropy(nn.Module):
    def __init__(self, ε:float=0.1, reduction='mean'):
        super().__init__()
        self.ε,self.reduction = ε,reduction

    def forward(self, output, target):
        c = output.size()[-1]
        log_preds = F.log_softmax(output, dim=-1)
        loss = reduce_loss(-log_preds.sum(dim=-1), self.reduction)
        nll = F.nll_loss(log_preds, target, reduction=self.reduction)
        return lin_comb(loss/c, nll, self.ε)

Assembling the Dataset

Because I’m using Colab Pro, I have to import my dataset each time I run the notebook. I was able to find the Food-101 dataset on AWS thanks to fastAI because ETH Zurich was no longer hosting the dataset on its website (https://vision.ee.ethz.ch/datasets_extra/food-101/). The code below downloads the tar file, extracts, renames and reassembles the images based on the data from train.txt and test.txt files. I prefer to keep all my train images in food-101/images/train and all my test images in food-101/images/test. I also move all of my meta files to food-101/meta.

# Removing the default sample_data directory that comes with every new Colab instance
!rm -rf sample_data

# Removes the existing food-101 dataset for a fresh, complete download
!rm -rf food-101 

# Downloads the food-101 dataset tar file
!wget https://s3.amazonaws.com/fast-ai-imageclas/food-101.tgz 

# Untars the tar file
!tar -xvf food-101.tgz 

# Removes the redundant h5 directory
!rm -rf food-101.tgz food-101/h5/

# Creates the meta, train and test directories inside food-101/images
!mkdir food-101/meta food-101/images/train food-101/images/test

# Moves all .txt and .json files to the meta directory
!mv food-101/*.txt food-101/*.json food-101/meta/ 

# Moves all test images to the food-101/images/test directory and renames them.
with open('food-101/meta/test.txt') as test_file:
  for line in test_file:
    name_of_folder = line.split('/')[0]
    name_of_file = line.split('/')[1].rstrip()
    Path('food-101/images/' + name_of_folder + '/' + name_of_file + '.jpg').rename('food-101/images/test/' + name_of_folder + '_' + name_of_file + '.jpg')

# Moves all training images to the food-101/images/train directory and renames them.
with open('food-101/meta/train.txt') as train_file:
  for line in train_file:
    name_of_folder = line.split('/')[0]
    name_of_file = line.split('/')[1].rstrip()
    Path('food-101/images/' + name_of_folder + '/' + name_of_file + '.jpg').rename('food-101/images/train/' + name_of_folder + '_' + name_of_file + '.jpg')

# Removes empty directories inside Food-101/images.
with open('food-101/meta/train.txt') as train_file:
  for folder in train_file:
    name_of_folder = folder.split('/')[0]
    if os.path.exists('food-101/images/' + name_of_folder):
      shutil.rmtree('food-101/images/' + name_of_folder)

Running shell commands to see if I’ve partitioned the train and test images correctly. There need to be 75,750 training images and 25,250 test images.

%cd /content/food-101/images/train/
!ls -1 | wc -l

%cd ../test
!ls -1 | wc -l

%cd /content/

/content/food-101/images/train
75750
/content/food-101/images/test
25250
/content

Building the Model

Because every image in the entire dataset comes labeled, I’ve decided to use the names of each image as classes for the classifier via regex. I am training the model with 20% of the training set as my validation set. For data augmentation, I am using fastAI’s .get_transforms() function because the transformations are done on the fly so there isn't a drastic increase in the dataset size, and the randomness is taken care of by the fastAI library. I am reducing each image to size 224 for better training. The batch size is 8 because I don't want to exceed the memory limit on CUDA (I've had issues with this when I was using larger batch sizes). I'm using .normalize(imagenet_stats) to easily normalize the data based on the RGB channels' stats from the ImageNet dataset.

np.random.seed(42)
batch_size = 8

path = 'food-101/images/train'
file_parse = r'/([^/]+)_\d+\.(png|jpg|jpeg)$'

data = ImageList.from_folder(path).split_by_rand_pct(valid_pct=0.2).label_from_re(pat=file_parse).transform(get_transforms(), size=224).databunch(bs = batch_size).normalize(imagenet_stats)

In my learner where I’ve added ResNet50 as my base architecture, I’m using accuracy for my metric and I’m measuring for top-1. I’m also applying Label smoothing to my model. Label smoothing helps train the model around misclassified data to improve its performance. While it causes the model to learn incorrectly by a lesser percentage, it also decreases the loss. When I relaxed confidence around the labels, I saw that the model was performing much better against the test set.

top_1 = partial(top_k_accuracy, k=1)
learn = cnn_learner(data, models.resnet50, metrics=[accuracy, top_1], loss_func = LabelSmoothingCrossEntropy(), callback_fns=ShowGraph)

Training

This is the training phase of my model. Before I run a set of epochs, I start by finding a learning rate. I added suggestion=True to approximate the interval I'll be selecting. In the below case, the optimal learning rate was around 1e-06.

learn.lr_find()
learn.recorder.plot(suggestion=True)

Based on the suggestion from the learning rate finder, I am getting a slice for the learning rate between 1e-06 and 1e-04. I’m running 5 epochs on this learning rate and saving the weights as stage-1. The model accuracy progresses in the first run. By the end of the 5th epoch, the model improved from 40.7% accuracy to 58.5%.

learn.fit_one_cycle(5, max_lr=slice(1e-06, 1e-04))
learn.save('stage-1')

Now I’m finding a new learning rate to run another set of epochs. I’m doing the same thing as above. One thing that I always do is that at the end of each 5 epoch cycle, I unfreeze the model. In this case, my learning rate was roughly around 1e-041.

learn.unfreeze()
learn.lr_find()
learn.recorder.plot(suggestion=True)

I ran another 5 epochs and saved it as stage-2. The model’s accuracy was increasing as fast as it was in stage-1 but this time, it intercepts the validation line.

learn.fit_one_cycle(5, max_lr=slice(1e-05, 1e-04))
learn.save('stage-2')

I wanted to see if I could squeeze out better accuracy. To do this, I recreated the data bunch object. I kept it the same as the earlier declaration except that here, I increased the image size from 224 to 512. I also wanted to load stage 2 to run the learning rate finder.

data = ImageList.from_folder(path).split_by_rand_pct(valid_pct=0.2).label_from_re(pat=file_parse).transform(get_transforms(), size=512).databunch(bs = batch_size).normalize(imagenet_stats)
learn = cnn_learner(data, models.resnet50, metrics=[accuracy, top_1], loss_func = LabelSmoothingCrossEntropy(), callback_fns=ShowGraph)
learn.load('stage-2')

learn.unfreeze()
learn.lr_find()
learn.recorder.plot(suggestion=True)

I ran another set of 5 epochs on the new learning rate. My last set of epochs gave me an accuracy score of 77.3% but after increasing the image size, I reached 84.2% accuracy. Training time increased significantly (from 11 minutes per epoch to 30 minutes) because I’ve increased the image size.

learn.fit_one_cycle(5, max_lr=slice(1e-05, 1e-04))
learn.save('stage-3')

Although the curve is starting to straighten, I wanted to run an additional set of 5 epochs to get the most accuracy out of my model so I ran a new learning rate finder.

learn.unfreeze()
learn.lr_find()
learn.recorder.plot(suggestion=True)

learn.fit_one_cycle(5, max_lr=slice(1e-06, 1e-04))
learn.save('stage-4')

It doesn’t look like I’m going to get better than 84.7% accuracy during training. If I were to run additional epochs, the graph would just oscillate.

Mispredictions

I wanted to see which images caused the most problems for the model. When it comes to food, there is a lot of visual overlap. For example, it isn’t weird for a model to be confused between steak with gravy and some pudding-like dessert due to the similarities in color, shape, texture, etc.

interp = ClassificationInterpretation.from_learner(learn)
interp.plot_top_losses(12, figsize=(15, 11))

interp.most_confused(min_val=5)

[('filet_mignon', 'steak', 26),
 ('chocolate_mousse', 'chocolate_cake', 24),
 ('steak', 'filet_mignon', 22),
 ('donuts', 'beignets', 19),
 ('prime_rib', 'steak', 19),
 ('beef_tartare', 'tuna_tartare', 15),
 ('dumplings', 'gyoza', 15),
 ('chocolate_mousse', 'tiramisu', 14),
 ('chocolate_cake', 'chocolate_mousse', 13),
 ('pork_chop', 'steak', 13),
 ('apple_pie', 'bread_pudding', 12),
 ('ice_cream', 'frozen_yogurt', 12),
 ('grilled_salmon', 'pork_chop', 11),
 ('sushi', 'sashimi', 11),
 ('baby_back_ribs', 'steak', 10),
 ('chocolate_mousse', 'panna_cotta', 10),
 ('ramen', 'pho', 10),
 ('tiramisu', 'chocolate_mousse', 10),
 ('tuna_tartare', 'beef_tartare', 10),
 ('bread_pudding', 'apple_pie', 9),
 ('breakfast_burrito', 'huevos_rancheros', 9),
 ('lobster_bisque', 'clam_chowder', 9),
 ('apple_pie', 'baklava', 8),
 ('chicken_quesadilla', 'breakfast_burrito', 8),
 ('hamburger', 'pulled_pork_sandwich', 8),
 ('tuna_tartare', 'ceviche', 8),
 ('baby_back_ribs', 'pork_chop', 7),
 ('chocolate_cake', 'tiramisu', 7),
 ('falafel', 'tacos', 7),
 ('filet_mignon', 'pork_chop', 7),
 ('huevos_rancheros', 'nachos', 7),
 ('huevos_rancheros', 'tacos', 7),
 ('ice_cream', 'chocolate_mousse', 7),
 ('pork_chop', 'filet_mignon', 7),
 ('pulled_pork_sandwich', 'hamburger', 7),
 ('steak', 'prime_rib', 7),
 ('beet_salad', 'foie_gras', 6),
 ('bruschetta', 'caprese_salad', 6),
 ('ceviche', 'tuna_tartare', 6),
 ('cheesecake', 'carrot_cake', 6),
 ('cheesecake', 'strawberry_shortcake', 6),
 ('chicken_quesadilla', 'tacos', 6),
 ('escargots', 'french_onion_soup', 6),
 ('falafel', 'crab_cakes', 6),
 ('foie_gras', 'pork_chop', 6),
 ('huevos_rancheros', 'croque_madame', 6),
 ('peking_duck', 'spring_rolls', 6),
 ('ravioli', 'gnocchi', 6),
 ('ravioli', 'lasagna', 6),
 ('ravioli', 'shrimp_and_grits', 6),
 ('shrimp_and_grits', 'risotto', 6),
 ('caprese_salad', 'greek_salad', 5),
 ('cheesecake', 'bread_pudding', 5),
 ('cheesecake', 'panna_cotta', 5),
 ('chicken_curry', 'gnocchi', 5),
 ('chicken_quesadilla', 'nachos', 5),
 ('club_sandwich', 'grilled_cheese_sandwich', 5),
 ('crab_cakes', 'falafel', 5),
 ('croque_madame', 'grilled_cheese_sandwich', 5),
 ('eggs_benedict', 'croque_madame', 5),
 ('french_fries', 'poutine', 5),
 ('garlic_bread', 'pizza', 5),
 ('greek_salad', 'caesar_salad', 5),
 ('grilled_cheese_sandwich', 'club_sandwich', 5),
 ('grilled_salmon', 'foie_gras', 5),
 ('guacamole', 'nachos', 5),
 ('gyoza', 'dumplings', 5),
 ('hummus', 'chicken_quesadilla', 5),
 ('nachos', 'tacos', 5),
 ('onion_rings', 'fried_calamari', 5),
 ('pulled_pork_sandwich', 'grilled_cheese_sandwich', 5),
 ('risotto', 'fried_rice', 5),
 ('risotto', 'shrimp_and_grits', 5),
 ('steak', 'pork_chop', 5),
 ('strawberry_shortcake', 'cheesecake', 5)]

When I look at the output of .most_confused(), I can see that the model's confusion was between food that has little to no physical distinction. For example, the biggest confusion was between filet mignon and steak. Filet mignon and steak are actually the same things. Steak is just a broad range of beef cuts and filet mignon is just the cut from the smaller end of the tenderloin.

It's usually my habit to include a confusion matrix as well but each time I tried to include one, the kernel would just die and all of my variables would be lost so I decided to go without it.

Testing

Now it's time to test the trained model. I created the new data bunch variable data_test. It is the same as above except that in the split_by_folder(), I've added the test images.

I chose learn.validate() as my accuracy method. The reason why is that learn.validate() is used if there is an existing validation set and validation sets are always labeled. The fastAI library takes the test sets blindly even if the test images have labels so I can't calculate the accuracy on that alone. It made more sense for me to create a new data bunch object where I include my test set like a validation set (split_by_folder(train='train', valid='test')) and compare my trained model against it.

path = '/content/food-101/images'

data_test = ImageList.from_folder(path).split_by_folder(train='train', valid='test').label_from_re(file_parse).transform(size=512).databunch().normalize(imagenet_stats)
learn = cnn_learner(data, models.resnet50, metrics=[accuracy, top_1],callback_fns=ShowGraph)
learn.load('stage-4')

The validation gives me a SoTA of 88.1% ≈ 88%

learn.validate(data_test.valid_dl)

[0.5012631, tensor(0.8809), tensor(0.8809)]

By:

Onur Andros Ozbek

Montreal, Quebec