Understanding ADOPT: A New Optimizer Beyond Adam

Optimization is at the heart of deep learning. When training neural networks, we aim to minimize a loss function that measures how well our model performs. Optimizers are algorithms that help us find the best parameters (weights and biases) that minimize this loss.

Let’s dive into the evolution of optimizers and understand why ADOPT is a significant advancement.

The Journey from SGD to ADOPT

Stochastic Gradient Descent (SGD)
The simplest optimizer is SGD, which updates parameters in the direction of steepest descent:

θ_t = θ_{t-1} — α∇L(θ_{t-1})

where α is the learning rate and ∇L is the gradient of the loss function.

However, SGD has limitations:
1. Fixed learning rate for all parameters
2. Sensitive to feature scaling
3. Can get stuck in saddle points
4. Slow convergence in ravines

Momentum-Based Methods
To address SGD’s limitations, momentum was introduced. It accumulates past gradients to maintain motion in consistent directions:

v_t = βv_{t-1} + (1-β)∇L(θ_{t-1})
θ_t = θ_{t-1} — αv_t

This helps:
1. Faster convergence in ravines
2. Better escape from local minima
3. Smoother optimization trajectory