Gradients, Hessians, and Derivative Tests: The Mathematical Backbone of Optimization

Optimization is the process of finding the best solution out of all possible solutions. The word “optimal” essentially means “best possible.” In mathematics, we solve optimization problems by finding the maximum or minimum of a function. In single-variable optimization, we find the extrema by computing the first-order derivative and then using the second derivative to determine whether it corresponds to a maximum or a minimum. But what if we are dealing with multivariable optimization? That’s where gradients, Hessians, and various derivative tests come into play.

Isn’t it amazing that we have powerful methods to determine the maxima and minima of complex higher-dimensional manifolds with precision and confidence !!

Single Variable Optimization:

Many of us intuitively think of the derivative at a point as the slope of the tangent line and the second derivative as representing curvature. Graphically, the second derivative measures the rate of change of the slope a negative second derivative means the slope is decreasing, while a positive second derivative means the slope is increasing. The sign of the second derivative is beneficial because it helps classify points as maxima or minima.

Multivariable Optimization:

Visualizing directional derivatives, cross-sections, and how Gradients & Hessians help find and classify local optima.

Multivariate function with multiple inputs and a single output, f(x,y) = sin(x)cos(y).

The derivative measures a function’s rate of change, but in higher dimensions, this rate depends not only on location but also on direction. To analyze the rate of change along a specific direction, we could slice the function with a plane and examine its cross-section. With some effort, we can derive an equation for this curve, which, being one-dimensional, allows us to compute derivatives as usual. However, repeatedly finding this curve for every direction can become tedious and complex. Fortunately, we don’t have to — this is where Gradients and Hessians come to the rescue, providing a powerful way to compute first and second derivatives in any direction effortlessly.

Just as first and second derivatives provide crucial insights into single-variable functions, their counterparts in multivariable functions hold distinct significance. In higher dimensions, derivatives not only measure the rate of change but also capture directional behavior and curvature across multiple axes. Understanding these derivatives — through gradients and Hessians.

The First Derivative is not a Number, it’s a Vector called The “Gradient”.

The Second Derivative is also not a Number, it’s a Matrix called “Hessian”.

Gradient

The gradient vector (∇f) of a function f(x) is a vector that points in the direction of the steepest ascent and contains the partial derivatives concerning each variable. It provides complete information about how changes in all directions at a given point.

Get Pradhum Thakur’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

∇f Or ∇Y is called the Gradient of Y, It is very common practice to represent the ∇Y by vector while implementing an optimization algorithm.

Formula of Gradient vector for function with n-dimensional input

Hessian

The second-order derivatives are used to get an idea of the shape of the graph for the given function. Whether it will be upwards to downwards based on concavity. The concavity of the given graph function is classified into two types: — Concave up and Concave down.

Formula of Hessian Matrix for function with n-dimensional input

CONCLUSION:

So if we have a point with a zero gradient and the second derivative is negative for every cross-section then this point will be a local maximum this means that the all the eigenvalues of Hessian matrix are negative and for local minimum case all the eigenvalues of Hessian matrix are positive.

Local Maximum Case

We already know that the second derivative of a function determines the local maximum (maxima) or minimum (minima), inflection point values. These can be identified with the help of below conditions:

1. If the Hessian matrix is positive (Positive eigenvalues of the Hessian matrix), the critical point is a local minimum of the function.

2. If the Hessian matrix is negative (Negative eigenvalues of the Hessian matrix), the critical point is a local maximum of the function.

3. If the Hessian matrix is indefinite (when it is not possible to conclude positive and negative eigenvalues of the Hessian matrix), the critical point is an inflection point.

Thanks for reading !!

References:

Mathematical Optimization

Visual intuition partial/directional derivative