The Security Researcher’s Guide to Mathematics

You can be a successful security researcher without knowing much about math. But if you want to see the matrix, you need to get math-pilled.

There is an old philosophical debate about whether mathematics is discovered or invented. As a kid, I wanted to do neither. Math felt abstract and disconnected from reality, a set of arbitrary rules to memorize. For a long time, I actively tried to get away with as little math as possible.

My perspective started to shift however when I began learning machine learning fundamentals. Later, diving into zero-knowledge proofs opened up an even broader world of mathematics. These days, I enjoy math for its own sake, but it also provides a deeper understanding of my daily work.

In this article, I’m introducing fields of math that, while not always strictly “essential,” are high-leverage for modern security researchers. Be warned however that I’ll take some detours into machine learning and physics.

Linear Algebra

At its simplest, algebra is the art of manipulating symbols under fixed rules. When we solve 2x + 3 = 11, the “algebra” is subtracting 3 and dividing by 2 to isolate x. In the nineteenth century, mathematicians like Gauss systematized how to solve such equations, leading to the concept of matrices and algorithms for manipulating them.

Linear algebra is the branch where operations stay linear (see below). Today, it underpins 3D graphics, recommendation engines, and machine learning. Large Language Models are essentially a massive stack of matrix multiplications with non-linearities mixed in.

What Does “Linear” Even Mean?

People sometimes use “linear” as shorthand for “simple,” but in math, it’s a structural constraint. A transformation L is linear if it satisfies two rules:

• Additivity: L(u + v) = L(u) + L(v)
• Homogeneity: L(c·v) = c·L(v)

In plain English: if you transform two things and add the results, you get the same answer as adding them first and then transforming. And if you scale something before transforming, it’s the same as transforming first and then scaling.

Think of it geometrically: if you have a grid, a linear transformation might rotate, stretch, or shear it, but the grid lines remain parallel and evenly spaced. It never curves or warps the space.

Vectors and Matrices: The Building Blocks

A vector is simply a list of numbers, such as v = (v₁, v₂, …, vₙ). You can think of it as a point in space or an arrow pointing in some direction. We usually treat vectors as column vectors in the vector space ℝⁿ (the set of all n-dimensional real number lists).

A matrix is a grid of numbers that transforms vectors. If A is a matrix and x is a vector, multiplying them (A·x) yields a new vector. Think of a matrix as a machine: you feed in a vector, and it spits out a transformed vector.

Let’s multiply a 2×2 matrix by a vector:

A = | 2   3 |      x = | 5 |
    | 4  -1 |          | 2 |

A·x = | 2×5 + 3×2    |   =   | 10 + 6 |   =   | 16 |
      | 4×5 + (-1)×2 |       | 20 - 2 |       | 18 |

Each row of A “dots” with x: multiply corresponding entries and sum. This operation is how every neural network layer, every 3D rotation, and every state transition in a Markov chain works under the hood.

Essential Operations

Dot product: ⟨x, y⟩ = xᵀy = Σᵢ xᵢyᵢ. This multiplies corresponding entries and sums them up. It measures how “aligned” two vectors are; if they point in similar directions, the dot product is large and positive. The length of a vector is ‖x‖ = √⟨x,x⟩ (the Euclidean distance from the origin).

Matrix multiplication: (AB)ᵢⱼ = Σₖ AᵢₖBₖⱼ. Composing two linear transformations (do A, then do B) is equivalent to multiplying their matrices together.

Identity and inverses: The identity matrix Iₙ has 1s on the diagonal and 0s elsewhere — it’s the “do nothing” transformation. A matrix A is invertible when there exists A⁻¹ such that AA⁻¹ = I. Inverting lets you “undo” a transformation.

Example: Token embeddings

A very prevalent application is in the embeddings used in modern machine learning. Training a large language model positions each token (word or word-piece) as a point in a high-dimensional space, nudging related tokens together and pushing unrelated ones apart.

The classic example: take the vector for King, subtract Man, add Woman, and the result lands eerily close to Queen:

King - Man + Woman ≈ Queen

This works because the model learned that “King” and “Queen” differ in roughly the same direction as “Man” and “Woman” — a direction we might call “gender.” We are doing “meaning calculations”, how awesome is that! You can build semantic search, clustering, or prompt-steering tooling by exploiting these geometric relationships.

Circuit Representations in Zero-Knowledge Proof Systems

Zero-knowledge proof systems often reduce verification to checking a bunch of linear equations. The key insight is that any computation can be encoded as a system of linear constraints over a finite field. This process is called “arithmetization.”

Rank-1 Constraint Systems (R1CS)

A common representation is a Rank-1 Constraint System (R1CS). We prove we know a secret solution vector s (called the “witness”) such that for many triples of vectors (a, b, c):

(s · a) × (s · b) = s · c

If all those equalities hold (using dot products), the program logic was followed correctly. The prover demonstrates knowledge of s without revealing it.

Example: Proving x³ + x + 5 = 35

Let’s say we want to prove we know x such that x³ + x + 5 = 35 (spoiler: the answer is x = 3).

Step 1: Flatten to quadratic gates

R1CS can only handle one multiplication per constraint, so we introduce intermediate variables to break down the cubic:

• v₁ = x × x (compute x²)
• v₂ = v₁ × x (compute x³)
• out = v₂ + x + 5 (final sum)

Step 2: Build the witness vector

s = [1, x, v₁, v₂, out] = [1, 3, 9, 27, 35]

The “1” at the start is a constant term that lets us encode addition and constants in our constraints.

Step 3: Encode as R1CS constraints

Each multiplication gate becomes (s·a) × (s·b) = (s·c):

Constraint       | a             | b             | c             | Check
─────────────────┼───────────────┼───────────────┼───────────────┼─────────────
v₁ = x × x       | [0,1,0,0,0]   | [0,1,0,0,0]   | [0,0,1,0,0]   | 3 × 3 = 9 ✓
v₂ = v₁ × x      | [0,0,1,0,0]   | [0,1,0,0,0]   | [0,0,0,1,0]   | 9 × 3 = 27 ✓
out = v₂ + x + 5 | [5,1,0,1,0]   | [1,0,0,0,0]   | [0,0,0,0,1]   | 35 × 1 = 35 ✓

The last constraint uses a trick: we multiply (v₂ + x + 5) by 1 to encode the addition as a “multiplication.”

When reviewing a Groth16 circuit, you’re checking that these constraint matrices correctly encode the intended computation. A missing constraint or wrong coefficient means the prover can cheat. For a more detailed introduction, see Vitalik’s classic QAP blog post and the RareSkills ZK Book.

Linearizing DeFi Invariants

DeFi invariants are almost always products or ratios raised to powers (Uniswap v2’s x·y=k, Balancer’s weighted geometric mean, Curve’s combination of both). We can unlock powerful analysis techniques by transforming them into linear equations that can then be solved with high school algebra, spreadsheets, or matrix operations (yes, successful security auditors use spreadsheets). The key technique is logarithmic linearization.

Logarithms have a magical property: they turn multiplication into addition.

ln(a × b) = ln(a) + ln(b)
ln(aⁿ) = n · ln(a)

This means any equation built from multiplication and exponentiation can be converted into addition and scalar multiplication, which is exactly what “linear” means:

Original Form          →    Linearized Form
─────────────────────────────────────────────────────────
x · y = k              →    ln(x) + ln(y) = ln(k)
xᵃ · yᵇ = k            →    a·ln(x) + b·ln(y) = ln(k)
(x/y)ᵃ · zᵇ = k        →    a·ln(x) - a·ln(y) + b·ln(z) = ln(k)

Example: Balancer-Style AMM

Automatic market makers maintain an invariant that can be linearized. Take a Balancer-style pool with reserves x = (xₐ, xᵦ, xᵧ) and weights w = (0.5, 0.3, 0.2). The invariant is:

xₐ^0.5 × xᵦ^0.3 × xᵧ^0.2 = k

This is a nonlinear equation with products and exponents everywhere. You can’t easily solve this in a spreadsheet. But watch what happens when we take the logarithm of both sides:

Starting equation:
  xₐ^0.5 × xᵦ^0.3 × xᵧ^0.2 = k

Take ln() of both sides:
  ln(xₐ^0.5 × xᵦ^0.3 × xᵧ^0.2) = ln(k)

Apply ln(a × b) = ln(a) + ln(b):
  ln(xₐ^0.5) + ln(xᵦ^0.3) + ln(xᵧ^0.2) = ln(k)

Apply ln(aⁿ) = n·ln(a):
  0.5·ln(xₐ) + 0.3·ln(xᵦ) + 0.2·ln(xᵧ) = ln(k)

Now substitute variables. Let vₐ = ln(xₐ), vᵦ = ln(xᵦ), vᵧ = ln(xᵧ):

0.5·vₐ + 0.3·vᵦ + 0.2·vᵧ = ln(k)

This is a linear equation in the v variables! In vector form:

w · v = c
where w = [0.5, 0.3, 0.2], v = [vₐ, vᵦ, vᵧ], c = ln(k)

Analyzing a trade

Suppose the pool starts at x = (1000, 800, 600). A trader wants to add 100 units of A and remove only B. We need to find Δᵦ such that the invariant holds.

The change in log-space must satisfy w · Δv = 0 (the invariant constant doesn’t change):

0.5 × ln(1100/1000) + 0.3 × ln((800-Δᵦ)/800) + 0.2 × 0 = 0

Solving step by step:

0.5 × 0.0953 + 0.3 × ln((800-Δᵦ)/800) = 0

0.0477 + 0.3 × ln((800-Δᵦ)/800) = 0

ln((800-Δᵦ)/800) = -0.159

(800-Δᵦ)/800 = e^(-0.159) = 0.853

Δᵦ ≈ 117.6

First-order approximation (for quick experimentation):

For small changes, ln(1 + δ) ≈ δ. This gives a simpler matrix form:

w · (Δx / x) = 0

[0.5, 0.3, 0.2] · [100/1000, -Δᵦ/800, 0]ᵀ = 0
0.5 × 0.1 + 0.3 × (-Δᵦ/800) = 0
Δᵦ ≈ 133.33

The linear approximation overestimates slightly, but gives a quick sanity check. Auditors use this to bound slippage or detect invariant violations during flash loan sequences.

Example: Perpetual Exchange Risk Engine

Perpetual futures exchanges maintain sophisticated risk engines that can be elegantly modeled using linear algebra. While implementations vary, the mathematical structure is universal.

The core components:

• Position vector p: One component per market (e.g., p = [+2 BTC, -5 ETH, +1000 USDC])
• Price matrix P: A diagonal matrix of current prices (diagonal means prices don’t interact — each asset has its own price on the diagonal, zeros elsewhere)
• Notional exposures: P·p gives the dollar value of each position
• Margin coefficient matrix C: Encodes maintenance margin requirements per market

The solvency test becomes a simple matrix inequality:

C · (P · p) ≥ m

where m is the minimum equity vector. In words: multiply positions by prices to get dollar values, apply margin coefficients, check if you’re above minimums.

Stress testing applies a shock vector s (e.g., “BTC drops 20%, ETH drops 30%”) via a sensitivity matrix S:

C · (P · (p + S·s)) ≥ m

If this fails, liquidations trigger.

Every step is a matrix-vector multiply with safety guards. By modeling the system this way, you can:

• Write the constraints in a spreadsheet and test edge cases
• Find positions that satisfy margin requirements individually but fail when combined
• Identify price movements that cascade into mass liquidations
• Check if rounding errors accumulate dangerously

Understanding this structure lets you audit liquidation logic and find edge cases where positions slip through the cracks.

Further reading for linear algebra

If you understandably fell in love with linear algebra after reading this, check out the following links next:

• Essence of Linear Algebra (3Blue1Brown) — Gorgeous visuals that make matrices and vectors intuitive
• MIT OpenCourseWare 18.06 — Gilbert Strang’s classic lecture series
• RareSkills ZK Book — Linear algebra prerequisites for ZK specifically
• RareSkills Uniswap v2 book
• RareSkills Uniswap v3 book

Non-Linearity: Modeling the complexities of the real world

We just learned about linear algebra and its uses. But the real world is messy and there’s many things that cannot be modeled with linear relationships.

For example, if neural networks were composed only of linear layers, they would collapse into a single giant matrix multiplication. Mathematically, applying matrices A then B sequentially (A × B × x) is the same as applying one new matrix C where C = AB. No matter how many layers you stack, if they’re all linear, the whole network is just one linear operation. It couldn’t learn anything complex.

The XOR Problem

Consider the XOR function (Exclusive OR). It outputs 1 only if the inputs are different:

Input   →   Output
──────────────────
(0,0)   →     0
(1,1)   →     0
(0,1)   →     1
(1,0)   →     1

Imagine plotting these on paper. You have two “0” dots at bottom-left and top-right, two “1” dots at top-left and bottom-right.

Now take a ruler and try to draw a single straight line that separates the 0s from the 1s.

You can’t. This is linear inseparability. A linear classifier fails here 100% of the time. To solve this problem, neural networks apply non-linear activation functions (like ReLU and Sigmoid) between linear layers. The most common is ReLU (Rectified Linear Unit):

f(x) = max(0, x)

It turns all negative numbers to zero and leaves positive numbers alone. This seems simple but changes everything!

The geometric intuition: A linear transformation can rotate or stretch paper, but keeps it flat. A non-linear function effectively folds the paper. If you crease along the diagonal where the 0s are, you can fold those corners away from the 1s. Now, in this folded 3D space, you can slice a plane between them.

Example: How a Neural Network Solves XOR

The key insight is that the hidden layer transforms the input space so that the classes become linearly separable. Let’s build a network that actually works.

The Strategy

XOR outputs 1 in exactly two cases: (0,1) and (1,0). We’ll create two hidden neurons, each detecting one of these cases:

• h₁ detects “x₁ is on, x₂ is off” → fires for (1,0)
• h₂ detects “x₂ is on, x₁ is off” → fires for (0,1)

Then the output is simply: if either hidden neuron fires, output 1.

The Network

Input layer: x = [x₁, x₂]

Hidden layer (2 neurons with ReLU activation):
  h₁ = ReLU(x₁ - x₂ - 0.5)    # Fires when x₁ > x₂ + 0.5
  h₂ = ReLU(x₂ - x₁ - 0.5)    # Fires when x₂ > x₁ + 0.5

Output layer:
  y = h₁ + h₂                  # Sum the detectors

Verification

Input   │ x₁-x₂-0.5 │ x₂-x₁-0.5 │  h₁  │  h₂  │ y=h₁+h₂ │ Target
────────┼───────────┼───────────┼──────┼──────┼─────────┼────────
(0,0)   │   -0.5    │   -0.5    │  0   │  0   │    0    │   0 ✓
(0,1)   │   -1.5    │    0.5    │  0   │ 0.5  │   0.5   │   1 ✓
(1,0)   │    0.5    │   -1.5    │ 0.5  │  0   │   0.5   │   1 ✓
(1,1)   │   -0.5    │   -0.5    │  0   │  0   │    0    │   0 ✓

With a threshold at 0.25, all outputs classify correctly.

The Geometric Picture

Original input space:              Transformed space (h₁, h₂):
                                   
    x₂                                 h₂
    ▲                                  ▲
  1 │ ●(0,1)     ○(1,1)              │ 
    │   class 1    class 0         0.5├───●(0,1)
    │                                 │
  0 │ ○(0,0)     ●(1,0)               │
    └──────────────► x₁             0 ○───────────●(1,0)──► h₁
        0          1                    (0,0)     0.5
                                        (1,1)
      
      No line separates                 Now linearly separable!
      ● from ○                          A line y = 0.25 works

The hidden layer “folds” the input space. Points (0,0) and (1,1) collapse to the origin, while (0,1) and (1,0) get pushed to different axes. Now a simple linear boundary separates the classes.

Abstract Algebra: The Math of Structure and Symmetry

Abstract algebra studies structures like groups, rings, and fields. It generalizes arithmetic beyond numbers to symmetry operations, matrices, or functions that you can combine under specific rules.

The field was born from attempts to solve polynomial equations. In the 1830s, Évariste Galois — a brilliant mathematician who died in a duel at 20 — realized that the symmetries of a polynomial’s roots formed an algebraic structure. He showed why a general fifth-degree polynomial has no formula using radicals, and in doing so, invented group theory.

Groups: The Algebra of Symmetry

A group is a set G with an operation · that satisfies four rules:

1. Closure: a·b is in G for all a, b in G (combining two elements gives another element in the set)
2. Associativity: (a·b)·c = a·(b·c) (parentheses don’t matter)
3. Identity: There exists e such that e·a = a·e = a (there’s a “do nothing” element)
4. Inverses: For each a, there exists a⁻¹ such that a·a⁻¹ = e (everything can be undone)

Classic example: integers under addition (ℤ, +). The identity is 0, and the inverse of n is -n.

But Why “Symmetry”?

The connection between groups and symmetry isn’t obvious from the integers example. To see it, consider a more geometric example: the symmetries of an equilateral triangle.

You can transform an equilateral triangle in ways that leave it looking identical: rotate it 120°, rotate it 240°, or flip it across any of three axes. Include “do nothing” and you have exactly 6 symmetry operations. These operations form a group:

• Closure: Doing one symmetry followed by another gives you another symmetry (rotating 120° twice is the same as rotating 240°)
• Associativity: The order of combining three operations doesn’t matter
• Identity: “Do nothing” leaves the triangle unchanged
• Inverses: Every rotation can be undone (120° clockwise undoes 120° counter-clockwise)

This is the dihedral group D₃ — the symmetry group of an equilateral triangle.

The integers under addition are also a symmetry group, just a more abstract one. Think of it as the symmetries of an infinite number line: adding 3 shifts every point right by 3, and this “shift” operation preserves the structure of the line (distances between points stay the same). Adding 0 is “do nothing,” and adding -3 undoes adding 3.

The key insight: Whenever you have a collection of transformations that preserve some structure, those transformations naturally form a group. Mathematicians discovered that the group axioms (closure, associativity, identity, inverses) are exactly what you need to capture “things you can do and undo while preserving structure.”

The Physics Connection: Noether’s Theorem

In the early 20th century, Emmy Noether proved one of the most profound theorems in history: every continuous symmetry of a physical system corresponds to a conservation law.

• Time symmetry (physics doesn’t change day to day) → Conservation of Energy
• Translation symmetry (physics is the same here and in space) → Conservation of Momentum
• Rotational symmetry (doesn’t matter which way you face) → Conservation of Angular Momentum

As it turns out, the fundamental forces of nature emerge from symmetries as well! Physicists write SU(3) × SU(2) × U(1) to describe the Standard Model of particle physics. Each group in that formula corresponds to a force:

• U(1) → Electromagnetism (carried by photons)
• SU(2) → Weak nuclear force (carried by W⁺, W⁻, and Z bosons)
• SU(3) → Strong nuclear force (carried by 8 types of gluons)

The number of force-carrying particles is determined by the mathematics of the group. U(1) has 1 generator, so electromagnetism has 1 force carrier (the photon). SU(2) has 3 generators, giving 3 weak bosons. SU(3) has 8 generators, hence 8 gluons.

Basically, physicists asked: “What if we require the laws of physics to remain unchanged under certain local transformations?” This requirement, called gauge invariance, mathematically forces the existence of force-carrying particles. The math dictates the physics.

Even gravity fits this pattern. General relativity emerges from requiring physics to be invariant under arbitrary coordinate transformations (diffeomorphism invariance). Spacetime curvature, which we experience as gravity, is the “gauge field” that preserves this symmetry.

This is arguably the deepest insight in modern physics: the structure of reality is not arbitrary. Forces, particles, and their interactions are inevitable consequences of abstract symmetry requirements. The universe appears to be, at its core, a theorem.

Exploits as symmetry breaking

Thinking about software exploits as symmetry breaking is a powerful mental model. A secure system relies on invariants — properties that must remain true no matter what valid transactions occur. An exploit is simply a transformation that breaks the symmetry.

When you audit a smart contract, you’re effectively checking if the “laws of physics” for that micro-universe hold up under all possible operations. Invariants can be checked via processes like input fuzzing and formal verification (see below).

Rings and Fields: Where Cryptography Lives

A ring is a set with two operations (+ and ×) where addition forms a group and multiplication is associative and distributes over addition. Think of it as “addition and multiplication, but maybe you can’t always divide.”

A field is a ring where you can also divide (every non-zero element has a multiplicative inverse). This is the algebraic structure where most cryptography lives.

Get Bernhard Mueller’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Integers modulo n (written ℤ/nℤ or ℤₙ) form a ring. They form a field only when n is prime — this is why cryptography loves primes.

Example: Arithmetic in ℤ₇

In ℤ₇ (integers mod 7), we do arithmetic and then take the remainder when dividing by 7:

Addition:  5 + 4 = 9 ≡ 2 (mod 7)
Multiply:  5 × 4 = 20 ≡ 6 (mod 7)
Inverse:   5⁻¹ = ? (find x where 5x ≡ 1 mod 7)
           5 × 3 = 15 ≡ 1 (mod 7), so 5⁻¹ = 3
Division:  4/5 = 4 × 5⁻¹ = 4 × 3 = 12 ≡ 5 (mod 7)

The extended Euclidean algorithm finds inverses systematically:

def extended_gcd(a, b):
    if b == 0:
        return a, 1, 0
    gcd, x1, y1 = extended_gcd(b, a % b)
    return gcd, y1, x1 - (a // b) * y1

def mod_inverse(a, n):
    gcd, x, _ = extended_gcd(a, n)
    if gcd != 1:
        raise ValueError("No inverse exists")
    return x % n

Homomorphisms: Structure-Preserving Maps

A group homomorphism is a function f: G → H between groups that respects the operation:

f(a · b) = f(a) · f(b)

In plain English: it doesn’t matter if you combine things before or after mapping — you get the same result. This “structure preservation” is cryptography’s secret weapon.

Homomorphisms for Hiding Information

Homomorphisms let you compute on hidden values without revealing them.

Pedersen Commitments:

Hide a value m by computing C = gᵐhʳ in a group where discrete log is hard:

• g, h are public generators (known starting points)
• m is your secret message
• r is a random blinding factor

The commitment is homomorphic:

Commit(m₁, r₁) × Commit(m₂, r₂) = g^(m₁+m₂) × h^(r₁+r₂) = Commit(m₁+m₂, r₁+r₂)

You can add commitments without revealing what’s inside! This is the foundation of confidential transactions and many ZK protocols.

Groth16 and PLONK proofs embed program constraints into group elements so that verifying equations reduces to checking that a homomorphism preserved structure.

At this point, take a pause and notice what just happened. “Structure preservation” is just another name for symmetry. The same idea appears in geometry, physics, and cryptography because reality, at every level, is built from symmetries!

The PlayStation 3 Hack

Sony famously learned abstract algebra the hard way. They reused a random nonce k in their ECDSA implementation.

How ECDSA works:

For private key d and message hash z:

1. Pick random k
2. Compute R = k·G (scalar multiplication on elliptic curve)
3. r = x-coordinate of R
4. s = k⁻¹(z + r·d) mod n

The signature is (r, s).

The vulnerability:

If you use the same k for two different messages z₁ and z₂:

s₁ = k⁻¹(z₁ + r·d)
s₂ = k⁻¹(z₂ + r·d)

Subtracting:
s₁ - s₂ = k⁻¹(z₁ - z₂)

Therefore:
k = (z₁ - z₂)/(s₁ - s₂)

Once you have k:
d = (s₁·k - z₁)/r

Sony’s private key could be calcated with simple algebra :)

Further reading for abstract algebra

• MIT OpenCourseWare 18.703 — Application-focused intro to groups, rings, and fields
• Quadratic Arithmetic Programs: from Zero to Hero — Vitalik Buterin’s tour of ZK-SNARKs
• Abstract Algebra: Theory and Applications — Free textbook covering two semesters

Number Theory: The Queen of Mathematics

Humanity has chased number theory puzzles for millennia. The Pythagoreans worshipped whole numbers and discovered irrationality by accident. Pierre de Fermat scrawled his infamous margin note about sums of powers. Carl Friedrich Gauss called number theory the “queen of mathematics” and turned it into a discipline with his Disquisitiones Arithmeticae.

For centuries, it was the poster child of pure math: Beautiful puzzles with no practical application. Then the 1970s arrived and public-key cryptography put prime numbers to work.

The One-Way Door: Factorization

The quintessential number theory problem in security is factorization. Given two large primes p and q, it’s trivial to compute n = p × q. But going the other way — finding p and q given only n — is computationally infeasible for large numbers.

RSA’s security relies on that one-way difficulty.

Worked Example: Toy RSA

Let’s build RSA with small primes to see the math:

Step 1: Choose primes
  p = 61, q = 53
  n = p × q = 3233

Step 2: Compute totient
  φ(n) = (p-1)(q-1) = 60 × 52 = 3120
  (The totient counts numbers less than n that share no factors with n)

Step 3: Choose public exponent
  e = 17 (must be coprime to φ(n), meaning they share no common factors)

Step 4: Compute private exponent
  d = e⁻¹ mod φ(n)
  17d ≡ 1 (mod 3120)
  d = 2753 (verify: 17 × 2753 = 46801 = 15 × 3120 + 1)

Public key:  (n=3233, e=17)

Private key: (n=3233, d=2753)

Encryption (message m=123):
  c = mᵉ mod n = 123¹⁷ mod 3233 = 855
Decryption:
  m = cᵈ mod n = 855²⁷⁵³ mod 3233 = 123

Why it works: Euler’s theorem says m^φ(n) ≡ 1 (mod n) for m coprime to n. Since ed ≡ 1 (mod φ(n)), we have ed = 1 + k·φ(n) for some k:

cᵈ = (mᵉ)ᵈ = m^(ed) = m^(1 + k·φ(n)) = m × (m^φ(n))^k ≡ m × 1^k = m (mod n)

The Discrete Logarithm Problem

Another staple: given gˣ mod p, find x. No efficient algorithm is known for large primes.

Example: Find x where 5ˣ ≡ 20 (mod 23)

Brute force:
  5¹ = 5
  5² = 25 ≡ 2
  5³ = 10
  5⁴ = 50 ≡ 4
  5⁵ = 20 ✓
So x = 5.

For 2048-bit primes, brute force takes longer than the age of the universe. This powers Diffie-Hellman key exchange.

Elliptic Curve Arithmetic

Elliptic curves over finite fields give us groups with hard discrete log but efficient operations.

An elliptic curve over field 𝔽ₚ: y² = x³ + ax + b (mod p)

Point addition: For P = (x₁, y₁) and Q = (x₂, y₂):

If P ≠ Q:
  slope m = (y₂ - y₁)/(x₂ - x₁) mod p
  
If P = Q (point doubling):
  slope m = (3x₁² + a)/(2y₁) mod p

Result R = P + Q:
  xᵣ = m² - x₁ - x₂ mod p
  yᵣ = m(x₁ - xᵣ) - y₁ mod p

All arithmetic happens inside the field 𝔽ₚ, including the division (which is really multiplication by modular inverse).

Fermat’s Little Theorem

Before we leave classical number theory, let me show you one of its most elegant results.

Fermat’s Little Theorem: If p is prime and a is not divisible by p, then:

a^(p-1) ≡ 1 (mod p)

That’s it. Raise any number to the power (p-1), divide by p, and the remainder is always 1.

Let’s verify with p = 7:

2^6 = 64 = 9×7 + 1 → remainder 1 ✓
3^6 = 729 = 104×7 + 1 → remainder 1 ✓
5^6 = 15625 = 2232×7 + 1 → remainder 1 ✓

Why does this work? Consider the set {1, 2, 3, 4, 5, 6} mod 7. Multiply every element by 3:

{3×1, 3×2, 3×3, 3×4, 3×5, 3×6} mod 7 = {3, 6, 2, 5, 1, 4}

We got the same set back, just reordered! This always happens when multiplying by something coprime to p. So:

(3×1)(3×2)(3×3)(3×4)(3×5)(3×6) ≡ 1×2×3×4×5×6 (mod 7)
3^6 × (1×2×3×4×5×6) ≡ (1×2×3×4×5×6) (mod 7)
3^6 ≡ 1 (mod 7)

The factorial cancels from both sides, leaving Fermat’s theorem.

Remember how RSA decryption works? We needed m^(ed) ≡ m (mod n). Euler’s theorem (the generalization of Fermat’s theorem to composite numbers) is exactly what makes this possible. The structure of prime numbers is the mathematical bedrock of public-key cryptography.

The Schwartz-Zippel Lemma: Why ZK Proofs Can Be Small

The Schwartz-Zippel lemma is one of my favorite theorems of all time. It makes it possible to verify an arbitrarily long proof in milliseconds!

The problem: Suppose I have two polynomials and want to check if they’re identical. The naive approach compares all coefficients — expensive for high-degree polynomials. But there’s a shortcut.

Schwartz-Zippel Lemma: Let P(x) be a non-zero polynomial of degree d over a field 𝔽. If you pick a random point r from 𝔽, the probability that P(r) = 0 is at most d/|𝔽|.

In other words: non-zero polynomials have very few roots relative to the field size.

Consider P(x) = x³ — 6x² + 11x — 6 = (x-1)(x-2)(x-3). This degree-3 polynomial has exactly 3 roots: {1, 2, 3}.

If we’re working in 𝔽₁₀₁ (integers mod 101), a random evaluation lands on a root with probability at most 3/101 ≈ 3%. One random check gives us 97% confidence the polynomial is non-zero. This transforms polynomial identity testing from O(d) coefficient comparisons to O(1) point evaluations:

Want to prove: P(x) = Q(x) for all x

Equivalent to: P(x) - Q(x) = 0 for all x

Test: Pick random r, check if P(r) - Q(r) = 0
If P ≠ Q, the difference polynomial has degree ≤ d,
so it has at most d roots out of |𝔽| possibilities.
Random r hits a root with probability ≤ d/|𝔽|.

With cryptographic field sizes (|𝔽| ≈ ²²⁵⁶), even degree-million polynomials give collision probability ≈ 1⁰⁶/²²⁵⁶ ≈ 0.

Modern ZK systems (Groth16, PLONK, STARKs) encode computations as polynomial constraints. A prover wants to convince a verifier that “I executed this computation correctly” without revealing the inputs.

The protocol typically works like this:

1. Encode computation as polynomials: The correct execution trace satisfies P(x) = 0 for all x in some domain
2. Prover commits to polynomials: Using cryptographic commitments that hide coefficients
3. Verifier picks random challenge r: After commitment (so prover can’t cheat)
4. Prover reveals P(r): Just one field element
5. Verifier checks P(r) = 0: If true, accept with overwhelming probability

By Schwartz-Zippel, a cheating prover who commits to the wrong polynomial will fail the check with probability ≥ 1 — d/|𝔽| ≈ 1.

Therefore, verifying a computation that took millions of steps reduces to checking a single point! This is why ZK proofs are “succinct” — the proof size and verification time are tiny compared to the original computation.

Further reading for number theory

• Cryptography I (Coursera) — Dan Boneh’s legendary course
• Moon math manual by Least Authority — explains all the math needed to understand zero-knowledge proofs

Mathematical Logic: The Limits of Proof

In the 1930s, Alan Turing and Alonzo Church tackled the foundations of computation from different angles. While Turing imagined a machine with a tape, Church invented a system of pure functions called Lambda Calculus. Remarkably, they’re equivalent — anything you can do with loops and state, you can do with recursive functions.

This led to the Curry-Howard Correspondence: “Proofs are Programs.” A mathematical proof corresponds to a computer program; a mathematical formula corresponds to a data type.

Gödel’s Incompleteness Theorem

Mathematical logic formalizes reasoning with symbols. By the 1930s, logicians hoped to capture all of arithmetic in a tidy, complete axiom system. Kurt Gödel dashed that dream with his incompleteness theorems.

In plain language: any consistent system strong enough to express basic arithmetic will contain true statements it cannot prove, and it cannot prove its own consistency from within.

The proof idea: Gödel encoded arithmetic statements as numbers (“Gödel numbering”), then used self-reference to construct a statement G that roughly says “G is not provable in this system.”

• If the system proves G, it’s inconsistent (G says it can’t be proved)
• If the system doesn’t prove G, then G is true but unprovable

Either way, completeness is impossible.

Turing and the Halting Problem

Alan Turing extended the story to computation. He modeled programs as Turing machines and proved in 1936 that no mechanical procedure can decide for every program whether it halts.

The proof: Suppose we had a halting oracle H(P, x) that returns true if program P halts on input x.

Build a program D that:

1. Takes program P as input
2. Asks H(P, P) — “does P halt when run on itself?”
3. If H says yes, loop forever
4. If H says no, halt immediately

Now run D(D):

• If H(D, D) = true, then D loops forever, so it doesn’t halt. Contradiction.
• If H(D, D) = false, then D halts immediately. Contradiction.

No such oracle can exist.

The halting problem is weirdly good news for hackers and defenders alike. It guarantees there will never be a push-button tool that certifies all software as perfectly safe. That motivates creative analysis, bug bounties, and layered defenses instead of complacent trust in static scanners.

Rice’s theorem generalizes this: every non-trivial semantic property of programs is undecidable. You can’t automatically determine if a program ever accesses an array out of bounds, leaks memory, or violates an invariant — for all possible programs.

Formal Verification

Formal verification aims to prove programs satisfy specifications, but Gödel and Turing remind us there’s no universal verifier. Practitioners work within carefully chosen logics and abstractions, proving partial properties while accepting undecidable cases.

Symbolic execution and model checking are how we wrestle undecidability in practice. Tools like Halmos, Mythril and hevm explore execution paths within resource limits, guided by SMT solvers.

In formal verification, we basically convert the analyzed code into math equations so we can then create mathematical proofs of certain properties.

Symbolic Execution: Turning Code into Algebra

The core idea of symbolic execution is deceptively simple: instead of running a program with concrete values like x = 5, we run it with symbolic values like x = α (some unknown). As the program executes, we build up algebraic expressions representing what each variable holds.

Consider a simple withdraw function:

function withdraw(uint256 amount) public {
    require(balances[msg.sender] >= amount);
    balances[msg.sender] -= amount;
    payable(msg.sender).transfer(amount);
}

When we symbolically execute this with amount = α (symbolic), here's what happens:

1. Initial state: balances[msg.sender] = β (also symbolic—we don't know the balance)
2. At the require statement: We record a path constraint: β ≥ α
3. After the subtraction: balances[msg.sender] = β - α
4. At the transfer: We note that α wei leaves the contract

The symbolic executor has now transformed this code path into a set of algebraic expressions:

Path constraint:     β ≥ α
Final balance:       β - α  
Amount transferred:  α

What the Algebraic Expressions Look Like

After exploring all paths, a symbolic executor has a collection of formulas. For a vulnerable contract, these might look like:

Path 1 (normal withdrawal):
  Constraints: β ≥ α ∧ α > 0
  Post-state:  balance' = β - α
  
Path 2 (reentrancy attack path):
  Constraints: β ≥ α ∧ attacker_is_contract
  Post-state:  balance' = β - α - α  (withdrew twice!)
  Violation:   balance' < 0  (underflow!)

The expressions use standard algebraic notation combined with bitvector arithmetic (since EVM uses 256-bit integers), array theory (for storage mappings), and uninterpreted functions (for hashes and external calls).

SMT Solvers: Finding the Exploit

An SMT (Satisfiability Modulo Theories) solver like Z3 takes these algebraic constraints and answers: “Is there an assignment of values that makes this formula satisfiable?”

For security analysis, we ask the solver: “Can you find inputs that violate this safety property?” For example:

from z3 import *

# Symbolic variables (256-bit integers like in the EVM)
balance = BitVec('balance', 256)
amount = BitVec('amount', 256)
attacker_balance = BitVec('attacker_balance', 256)
solver = Solver()

# Path constraints (what must be true to reach this point)
solver.add(balance >= amount)          # require passed
solver.add(amount > 0)                 # non-zero withdrawal

# The vulnerability condition we're hunting for
# Can the attacker end up with more than they started with?
solver.add(attacker_balance + amount > attacker_balance + balance)

if solver.check() == sat:
    model = solver.model()
    print(f"Exploit found!")
    print(f"  balance = {model[balance]}")
    print(f"  amount = {model[amount]}")

When the solver returns sat (satisfiable), it provides a counterexample, i.e. a set of concrete values that break the property. This counterexample is the exploit. The solver has effectively reverse-engineered the attack inputs from the algebraic constraints.

From Theory to Tooling

Modern smart contract security tools chain these steps together:

1. Parse the contract bytecode or source
2. Symbolically execute to build path constraints
3. Encode security properties (no reentrancy, no overflow, no unauthorized access)
4. Query the SMT solver for violations
5. Output concrete exploit transactions if found

Released in 2017, Mythril was one of the first tools implementing this technique for EVM smart contract. Building on Mythril, Scrooge McEtherface (presented at DEF CON 2019) takes it a step further: It automatically generates working exploit transactions that extract ETH from vulnerable contracts.

Newer tools like Halmos integrate symbolic testing directly into the Foundry development workflow. Instead of writing separate specifications, you write tests with symbolic inputs:

function check_withdraw(uint256 amount) public {
    uint256 balanceBefore = token.balanceOf(address(this));
    token.withdraw(amount);
    uint256 balanceAfter = token.balanceOf(address(this));
    
    // This assertion is checked for ALL possible values of amount
    assert(balanceAfter <= balanceBefore);
}

Halmos explores all ²²⁵⁶ possible values of amount symbolically, not by running them individually, but by building constraints and querying the solver. If any value breaks the assertion, Halmos reports a counterexample.

Further reading for mathematical logic

• Stanford Encyclopedia: Gödel’s Incompleteness Theorems
• Stanford Encyclopedia: The Halting Problem
• Gôdel, Escher, Bach by Douglas Hofstadter

Other Fields Worth Exploring

While the sections above cover the mathematics most directly applicable to smart contract auditing and cryptography, several other fields are worth knowing about.

Calculus: The Mathematics of Change

Zeno of Elea proposed a famous paradox: to walk across a room, you must first walk halfway, then half of that, then half again. Since there are infinite halves, you should never reach the other side. Mathematics solved this with calculus — the study of limits and infinite series. We can prove that an infinite sum of ever-smaller quantities converges to a finite number: 1/2 + 1/4 + 1/8 + … = 1 exactly.

Calculus gives us two fundamental tools: derivatives measure how sensitive a function is to small input changes (critical for gradient-based attacks on ML models and understanding numerical stability), while integrals measure accumulation over time (used in side-channel analysis where tiny power leaks are summed across thousands of traces until the signal emerges from noise). If you venture into adversarial machine learning, differential privacy, or hardware security, calculus becomes essential. For pure smart contract work, you can often get by without it, but understanding limits helps when reasoning about precision loss in repeated operations.

Discrete Mathematics: Graphs, Combinatorics, and Counting

Graph theory models relationships between objects — vertices connected by edges. In security, graphs are everywhere: network topologies (can an attacker pivot from the web server to the database?), smart contract call graphs (which functions can call which? are there cycles indicating reentrancy risk?), control flow graphs in binary analysis (which basic blocks dominate the path to a vulnerability?), and state machines (what sequences of transactions are possible?). The adjacency matrix representation connects graph theory back to linear algebra: if A is the adjacency matrix, then (Aᵏ)ᵢⱼ counts the number of length-k paths between nodes i and j.

Combinatorics — the mathematics of counting — helps analyze the size of search spaces (how many possible keys? how many execution paths?), understand birthday attacks and collision probabilities, and reason about the complexity of brute-force versus clever attacks. When someone says “256-bit security,” they’re making a combinatorial argument about the infeasibility of exhaustive search. These fields overlap heavily with algorithm analysis and complexity theory, which underpin our understanding of what makes cryptographic problems “hard.”

Bringing It All Together

These areas of math connect directly to what we do in security:

• Linear algebra lets you read ZK circuits, understand AMM invariants, and analyze neural network vulnerabilities
• Abstract algebra gives you the language of cryptography — groups, fields, homomorphisms — and explains why the PS3 hack worked
• Number theory is the foundation of RSA, Diffie-Hellman, and elliptic curves, with attacks lurking in every implementation detail
• Logic reminds us that perfect verification is impossible, which paradoxically makes security more interesting, not less

If you’re excited about the convergence of math, coding, and security, there has never been a better time to dive in. Free courses, videos, and welcoming communities are everywhere. Whether you start by visualizing a matrix with 3Blue1Brown or by coding your own implementation of Groth16, you’re in for an eye-opening experience.