Gradients and Gradient Tape — Part 1

Learn how TensorFlow used Gradient Tape to implement automatic differentiation. Essential to unbox what happens during backpropagation!

You can also read it here.

Gradients — Introduction

Mathematically, at the core of Neural Networks lies a complex and challenging curve fitting problem. Curve fitting is an optimization problem and our objective is to find the parameters of a curve that is the best fit for the dataset. This is one of the most important steps in any Neural Network and the inability to find the best fitting curve can have a huge impact on the end result.

The topic we’ll be exploring today is quite vast so I’ve divided it into different parts. The first part will cover the basics of gradient computation followed by some simple examples of Tensorflow’s Gradient Tape.

Before deep diving into the specifics of Gradient tape, let us build an intuitive understanding of gradients.

In simple terms Gradient is slope.

Line in 2D

The figure above displays the points on the 2D plane for line y = 0.5*x+2

Here the slope of the line (gradient) is 0.5

For curves, the gradient at any point is the derivative of the function f(x) at a particular x. The direction of slope is dependent on the…