Featured

Partial Derivatives and the Hessian — A Simple Guide

Calculus is an essential tool for understanding change, and partial derivatives play a key role when dealing with functions of multiple variables. In this article, we’ll explore partial derivatives and the Hessian matrix in a simple and accessible way.

Prerequisites before reading on:

• Know what differentiation is and how to compute basic derivatives
• Know the basics of matrices and eigen decomposition

What Are Partial Derivatives?

Imagine you’re climbing a hill, and you want to know how steep it is. But here’s the catch: the hill slopes differently depending on the direction you’re walking. A partial derivative tells you how steep the hill is in a specific direction.

For a function of two variables, like f(x, y), the partial derivatives are:

• ∂f/∂x: This tells you how f changes if you move along the x-axis, keeping y constant.
• ∂f/∂y: This tells you how f changes if you move along the y-axis, keeping x constant.

For example, if f(x,y) = x² + y²:

• ∂f/∂x = 2x: This measures the slope in the x-direction.
• ∂f/∂y = 2y: This measures the slope in the y-direction.

These partial derivatives are like looking at the hill from two different viewpoints.

Second-Order Partial Derivatives

Second-order partial derivatives look at how the rate of change itself changes. There are four second-order derivatives for a function f(x, y):

1. ∂² f/∂x²: How ∂f/∂x changes as x changes.
2. ∂² f/∂y²: How ∂f/∂y changes as y changes.
3. ∂² f/(∂x∂y): How ∂f/∂x changes as y changes.
4. ∂² f/(∂y∂x): How ∂f/∂y changes as x changes.

Interestingly, for most well-behaved functions, ∂² f/(∂x∂y) = ∂² f/(∂y∂x).

The Hessian Matrix

Now that we understand second-order partial derivatives, let’s combine them into something called the Hessian matrix. This matrix is a compact way to organise all the second-order information about a function.

Get Mehul Chourasia’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The general form for the Hessian matrix is:

Where n is the number of parameters the function has.

For a function f(x,y), the Hessian matrix is:

Why is this useful? The Hessian tells us about the curvature of a function at a point:

• If all eigenvalues of the Hessian are positive, the function has a local minimum.
• If all eigenvalues are negative, the function has a local maximum.
• If the eigenvalues have mixed signs, the function has a saddle point.

Let’s see why this is the case:

Curvature measures how a function changes as you move away from a point in a given direction. Mathematically, this is tied to the second derivative in that direction. For a symmetric matrix like the Hessian:

1. The eigenvectors v represent the principal directions of curvature (the directions in which curvature is measured most distinctly).
2. The eigenvalues λ​ represent the magnitude of curvature along these directions:

• λ > 0: Curves upwards.
• λ < 0: Curves downwards.
• λ = 0: No curvature (flat).

This is why eigenvalues correspond directly to the curvature: they tell us how the function bends in each eigenvector’s direction.

Intuition for the Hessian

Imagine you’re standing on a surface. The Hessian captures how the surface bends around you. If the bending is upwards (like a bowl), you’re at a minimum. If it bends downwards (like an upside-down bowl), you’re at a maximum. If it bends in mixed ways (like a saddle), you’re at a saddle point.

Why Do We Care?

Partial derivatives and the Hessian matrix are everywhere in science and engineering:

• Optimisation: When designing a machine or training a neural network, we often need to minimise or maximise a function. The Hessian helps us understand the nature of critical points.
• Economics: Partial derivatives describe how changes in one variable affect another in a system.

Final Thoughts

Understanding partial derivatives and the Hessian doesn’t have to be intimidating. Think of partial derivatives as measuring slopes in specific directions and the Hessian as a tool for analysing curvature. With these concepts, you’re well-equipped to tackle problems involving multivariable functions.