Optimizers!! bang how does it work mathematically ??

we know what Optimizers do and how it improvises model accuracy and reciete loss.

The definition, overview understanding, types, importance are all known.

But how does it work ?

let’s take an example,

2 * 3 = 6

x1 = 2, x2 = 3 are produced to a simple neural network with 2 hidden layers.

Random weights are assigned to every edge. Bias equal to 0.9.

Each node calculates,

h = w * x + b

this is the value of each successive node. x value of the previous node, w weight and b is bias. An activation function determines how ‘good’ the value is to take forward. We’ll use the ReLU activation function, it says — if output is negative reset to 0 or pass positive itself.

ReLU = max(0, x)

loss is calculated using actual y and predicted y expelled from the last node. let us take Squared error(SE)

SE = (Y — y) ^ 2

Optimizers help reduce these errors, by tweaking weights and bias at each node.

then,

1. What backpropagation does?
2. Isn’t backpropagation also contributes to reduce loss by changing params?
3. what is the ideological difference between backpropagation and optimizers?

We have answers for these..

let us continue with the above example.

Point loss value in a graph.

image

Get Kowshik T G’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Gradient is calculated using partial differentiation. Starting from the last node, dependency is related to its precedence entity. 3 different entities are differentiated with different variables.

a. Predicted value with the previous node, equation —

L = (Y — y) ^ 2

partial differentiation on both side

dL/dy = 2(Y — y)

b. next node formula, equation —

z = w * x + b

partial differentiation on both side

dZ/dw = x

dZ/dx = w

dZ/db = 1

c. activation function(considering ReLU), equation —

h = max(0, z)

partial differentiation on both side

dh/dz =

1, if z=positive

0, is z= negative

Backpropagation tells us the direction and magnitude of the local loss change, this is where chain-rule comes in. If there are value-dependencies between a dependency then, later entities incorporate with them.

example, in a network —

h1 → h2 → h3 → y

we have dy/dh3 = a, dh3/dh2 = b, dh2/dh1 = c, if we want to calculate dy/dh1 without knowing any dependency value, then we do something like

dy/dh1 = dy/dh3 * dh3/dh2 * dh2/dh1

resulting dy/dh1 = a*b*c

This chained dependency is called chain-rule.

Below is a notebook comprising how optimizers work. From building simple NN , calculating loss, and visualizing how optimizers change loss with change in params.

It incorporates markdown files explaining each step and some important line of code.

You’ll also find how each optimizer( SGD, SGD with Momentum, RMSProp, Adam) works, and results.

https://colab.research.google.com/drive/14_olwiDobgLGY2aLXZEhH20NzgEKyvbd?usp=drive_link

I guess you got a major part of how things work.

please try building one!!