Optimizers in Machine Learning and DL
Training a model, includes parameters that shape every input data. And training does help model learn the difference and similarity between data parameters.
More or less could affect learning and can change predictions.
Before explaining optimizers, can you think how to evaluate model accuracy!! Yeah, calculating the loss of the model. Loss —The difference in predicated and actual value for the same input.
We have different loss functions, Mean squared loss, log loss, hing loss, root mean squared loss..
Can we say that, tailing off loss value can increase model accuracy? Yes.
Optimizer does it. Change in model parameters (such as biases and weights) could significantly reduce loss with respect to parameter difference. Examples include, RMSprop, Adam, Ada, SGD,.
Those exact parameters are required to gain such confidence.
Optimizers calculate every possible way to increase the effectiveness of model accuracy, by finding those exact parameters…
The most popular method is
Gradient Descent —
It adjusts parameters to effectively reduce loss value and move towards Global minima. Refer below figure.
The incremental step is calculated using equation(1) to reduce cost with respect to updated weight, pathing to Global minima point.
θ (k+1) =θ k −α∇J(θ k) — — — — (1)
here, θ(k+1) is updated parameter,
θk is current parameter,
α is learning rate,
∇J(θk) is the gradient of that cost it is experiencing at.
GD with Armijo Goldstein —
We do have Gradient Descent with Armijo Goldstein method, where we update parameters with a given step size, which will be quick compared to iterative method, Gradient Descent.
Stochastic Gradient Descent —
Stochastic Gradient Descent is a type in Gradient Descent, where we update parameters by iterating on each training data, while normal GD uses the entire dataset on one go.
Below figure is a graph of SGD that shows meeting global minima is its aim by changing params.
Mini-batch GD —
Mini-batch GD is a type of GD where, specific training batch size is defined on each iteration and calculates it’s mean, quicker compared to SGD.
Below figure expels mini-batch GD averaging batch loss and pathing towards reduced loss.
SGD with Momentum —
SGD with Momentum reduces noise and accelerates convergence thus reducing fluctuations, increasing confidence likely. It iterates on every input one-by-one with an accelerated speed of reducing noise from predictions.
Get Kowshik T G’s stories in your inbox
Join Medium for free to get updates from this writer.
The updated value in the next iteration leads to lower loss quicker as per given equation(2).
v (t+1) =β∗v t +(1−β)∗∇J(θ t ) — — (2)
β specifies Momentum coefficient
and model params are updated as, equation(3).
θ(t+1)=θt−α∗v(t+1) — — — — — (3)
The figure below shows convergence in the loss threshold and narrowing down.
AdaGrad —
Adaptive GD adapts the learning rate for each param by using the accumulated sum of squared gradients, allowing more efficient and stable training.
As every other Optimizer, calculates loss -> updates params -> check for updates loss, and iterates. AdaGrad initializes a running sum of squared gradients for each parameter additionally. And adjusts the learning rate for each param based on the accumulated sum of squared gradients, instead of using a fixed rate.
The figure below specifies Learning rate calculation. (source — gfg)
η , is a constant global learning rate,
Gt is the sum of squared gradients for a given parameter up to time step t,
ϵ is a constant to avoid division by zero.
RMSProp —
RMSProp(Root Mean Squared Propagation) improves Adagrad by using an exponentially decaying average of squared gradients, instead of accumulating them, preventing the learning rate from shrinking it too quickly..
The figures below show equations to update weights.
AdaDelta —
AdaDelta improves Adagrad to avoid excessive accumulation of past gradients by using moving averages, and leading to more stable & consistent updates.
Adam —
Did you think, that combining AdaGD and SGD with Momentum, robust convergence and adapts to newer learning rates at each iteration, making it more effective and optimal than each individual.
Adam — Adaptive Moment Estimation, uses moving average of squared gradients (adaptive learning rate) with provided fast and stable convergence. Helps reduce narrow down path with accelerated learning leading to reduce loss. Works well across a wide range of tasks and models.
It is widely used in Deep Learning.
The figure below states all equations with a combined equation to calculate update weight using Adam.
I guess you somewhat understood what Optimizers are and their types with the right usage.
I thank GFG for their explanation on Optimizers. Image(graph, equation) Source — GFG
Refer https://www.geeksforgeeks.org/machine-learning/intuition-behind-adagrad-optimizer/ for info..
