Blog 2

BUILD AN AI TO BEAT THE GAME OF BREAKOUT

Introduction :

Breakout is an arcade game developed and published by Atari, Inc and released on May 13, 1976. It was conceptualized by Nolan Bushnell and Steve Bristow, influenced by the seminal 1972 Atari arcade game Pong, and built by Steve Wozniak.

Breakout was the basis and inspiration for certain aspects of the Apple II personal computer. In 1978, the game was ported to the Atari 2600 and a sequel was made, Super Breakout, which four years later became the “pack-in game” for the Atari 5200 console.

In Breakout, a layer of bricks lines the top third of the screen and the goal is to destroy them all. A ball moves straight around the screen, bouncing off the top and two sides of the screen. When a brick is hit, the ball bounces back and the brick is destroyed. The player loses a turn when the ball touches the bottom of the screen; to prevent this from happening, the player has a horizontally movable paddle to bounce the ball upward, keeping it in play.

Implementation :

Let’s first see the big picture. The A3C algorithm contains several key concepts that all make it work. Multiple separate environments are run in parallel, each of which contains an agent. The agents however share one neural network. Samples produced by agents are gathered in a queue, from where they are asynchronously used by a separate optimizer thread to improve the policy. The diagram below shows how these components work together

We will implement four classes — Environment, Agent, Brain, and Optimizer. Two of these are thread classes — Environment and Optimizer.

Model:

Now that we know how to play Breakout (or any other Atari game for that matter), we can get back to estimating our Q-function. As a reminder:

Q(s, a) = r + γ maxₐ’(Q(s’, a’))

Bellman’s equation

Our algorithm will play the game, each step saving the initial state, the action it took, the reward it got, and the next state is reached, and this data will be used to train a neural network with Keras.

The input of our neural network will be an initial state (ie a stack of 4 preprocessed frames), and its output will be its estimate of r + γ maxₐ’(Q(s’, a’)), where r is the reward that was obtained when playing, γ is our discount rate (0.99), and Q(s’, a’) comes from predicting the Q function for the next state using our current model. That’s right, in some sense our neural net is trying to predict its own output.

Coding/Working :

For Atari games, since they are much bigger than a simple mouse maze, it’s hard to use a Q-table for each action and pixel. It’s simply inefficient and not scalable. For more complex games, we use neural networks to make it easy and more efficient to learn and get better at the game. With these neural networks, we significantly reduce the number of possibilities from billions to only millions. This is done by the neural network approximating Q-values for each action, given a state.

Preprocessing :

After we have done that we can stack 4 frames. Why do we stack 4 frames? Well, imagine a picture of 2 cars facing each other on the road. From a single image, you can’t tell whether the cars are parked or about to crash, but if you’re given a few frames of it from a video you can easily tell if the cars are moving or not. Similarly, it’s easier for our DQN to make assumptions.

The 4 frames received are then processed by convolutional networks. These convolutional layers allow you to exploit some spatial properties across those frames.

Get sai varun gandla’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The next part of our network is Experience Relay. It helps us with 2 important things:

• reducing correlated data
• Better use of previous experiences

If we train the network in sequential order, we risk our agent being influenced by the effect of correlation. Since not much happens from frame to frame the network’s weights and bias change influencing the agent’s actions.

By sampling from the replay buffer at random, we can break this correlation. This prevents action values from oscillating or diverging catastrophically.

Essentially, the sample transitions are stored, which will then be randomly selected from the “transition pool” to update the knowledge.

Implementation :

Now after resolving the environmental problem we started implementing the brain of the project which is the A3C algorithm, so now we have to create a convolutional layer ->pooling layer-> flattering layer

And in the convolutional neural network, we have to implement the A3C algorithm with 3 Agents, we can have more than three Actors as of now we are just creating the 3 Agent to working simultaneously by using the SHARED MODEL.

we came to know that this feature extraction works on the principle of taking pixels from each movement for your better understanding i have enclosed the image below,

as we see in the above picture we can see that the next move will be left downwards but the AI doesn't know what to do, it will only come to know that it is moving left downwards after many softmax functions.

A3c brain is the most important of the project, this contains all A3C algorithms and the LSTM layer implemented.

Issues faced :

so this is where we found the problem that it is taking much time to think(train) and make the correct decision

so here comes the idea to the problem, what if we use the LSTM layer?

so finally the LSTM layer has solved the problem because the process of the LSTM layer was to save the previous move of the input in the hidden layers itself, now there won't be a problem of making the prediction of the ball where is going, all these things are done in the hidden layer itself, this makes the algorithm works way better than the others.

The video of the working model of our project is to be shown below :

Conclusion :

It’s been a model/project we have tried to implement using various platforms to beat the game breakout using AI.