AXIS: Dot Product - The Tiny Operation That Powers Transformer Attention
Part of the AXIS series — AI eXplained through Intuitive Systems
When people first learn transformers, the conversation quickly jumps to scary-sounding words:
embeddings, attention, Q/K/V, softmax, multi-head attention, token similarity.
But under all of that, one small mathematical operation keeps showing up again and again:
the dot product.
The dot product is one of those ideas that looks simple on the surface:
But this tiny formula has a much deeper meaning.
It answers a very important question:
How much does one vector point in the direction of another vector?
That question is not just geometry fun fact. That question becomes extremely important in AI.
Because in transformers, words, tokens, images, and concepts are all represented as vectors. So when the model asks, “How related is this token to that token?” it often starts with a dot product.
Let’s build the intuition from first principles.
1. What exactly is the dot product?
The dot product is an operation that takes two vectors and returns one number. That number tells us how aligned those two vectors are.
Suppose we have two vectors:
A = (x_1, y_1), and B = (x_2, y_2)
The algebraic definition of dot product is:
A.B = x_1*x_2 + y_1 * y_2
(* means simple multiplication above)
So if:
A = (2, 3), B = (4, 5)
A dot B = A.B = 2 * 4 + 3 * 5 = 8 + 15 = 23
But the calculation alone does not explain the meaning.
The deeper meaning is:
Dot product measures directional agreement.
If two vectors point mostly in the same direction, their dot product is large and positive.
Given a bunch of vectors and their dot products figuring out which ones are similar boils down to comparing their dot product. If A.B = 23 and A.C = 52 then A and S are more similar than A and B. Why? It’d become clearer by the end of this article.
If they are perpendicular, their dot product is zero.
If they point in opposite directions, their dot product is negative.
That is why the dot product is both an algebraic operation and a geometric signal.
2. The algebraic way: coordinate-by-coordinate agreement
The algebraic way is the one most people learn first. And then there is geometric way. The calculation from first principles is given in the attached image for the folks who want to understand the math behind in just one picture
Both algebraic and gemoetric calculations and their intuitions have been covered below.
For two vectors:
A.B = x_1*x_2 + y_1 * y_2
This says:
Multiply the x-parts together, multiply the y-parts together and then add them.
In higher dimensions, it works the same way.
If:
A = (a_1, a_2, a_3, …, a_n)
B = (b_1, b_2, b_3, …, b_n)
then:
A.B = a_1*b_1 + a_2*b_2 + a_3*b_3 + … + a_n*b_n
So the dot product is basically a weighted agreement score across dimensions.
Each dimension contributes to it.
If both vectors have large values in the same dimension, that dimension contributes a lot.
If one is positive and one is negative, that dimension subtracts from the score.
If one of them is close to zero, that dimension does not contribute much.
That is the algebraic intuition:
The dot product adds up how much two vectors agree dimension by dimension.
This is already useful in machine learning.
Because embeddings are just high-dimensional vectors.
A word embedding may not have just 2 dimensions. It may have 768, 1,024, or several thousand dimensions. OpenAI’s GPT-2 small used 768 dimensions, GPT-2 medium used 1,024
Each dimension is not usually human-readable, but together they encode meaning.
So when we take the dot product of two embeddings, we are asking:
Across all these hidden dimensions, how much do these two things agree?
3. The geometric way: length, angle, and alignment
There is another way to calculate dot product:
A.B = |A||B|cos(theta)
|A| is the length of vector A.
|B| is the length of vector B.
“theta” is the angle between the two vectors.
This version says something very intuitive:
Dot product depends on how long the vectors are and how much they point in the same direction.
The key term is:
cos(theta), Cosine tells us about direction.
When two vectors point in the same direction:
theta = 0, cos 0 = 1
A.B = |A| |B|
Maximum alignment.
When two vectors are perpendicular:
theta = 90, cos 90⁰ = 0
A.B = 0
No directional overlap.
When two vectors point in opposite directions:
theta = 180⁰
cos 180⁰ = -1
A.B = -|A| |B|
Strong disagreement.
This is the geometric intuition:
The dot product measures how much one vector aligns with another vector.
Or even more visually:
It measures the projection (or shadow) of one vector on another, scaled by the other vector’s length.
4. Why are the algebraic and geometric versions exactly the same?
At first, these two formulas look like two different ideas.
The algebraic version says: A.B = x_1x_2 + y_1y_2
The geometric version says: A.B = |A| |B| cos(theta)
One talks about coordinates.
The other talks about length and angle.
So why are they the same?
This is where the derivation in the image becomes important. Showing it below again:
That is the bridge.
The coordinate formula and the angle formula are not two separate truths.
Get Sanjiv’s stories in your inbox
Join Medium for free to get updates from this writer.
They are the same truth seen from two different perspectives.
5. Intuition behind the algebraic way
The algebraic way says:
A.B = x_1x_2 + y_1y_2
This feels mechanical, but it has a beautiful intuition.
Think of each vector as a profile.
For example, suppose we represent a movie using dimensions like:
( action, comedy, romance, suspense, emotional depth)
And suppose we represent a viewer’s preference using the same dimensions.
A movie vector may look like:
(8, 2, 1, 7, 4)
A user preference vector may look like:
(9, 1, 0, 8, 3)
The dot product compares them dimension by dimension.
Action matches action.
Comedy matches comedy.
Suspense matches suspense.
Then it adds all those matches into one score.
So algebraically, the dot product is:
A total compatibility score across matching dimensions.
This is why it is so useful in AI.
The model may not have dimensions like “action” or “romance” in a human-readable way, but the principle is similar.
Each dimension carries some learned signal.
The dot product asks:
Where both vectors are strong in the same learned directions, how much total agreement do we get?
That is the algebraic view.
6. Intuition behind the geometric way
The geometric way says:
A.B = |A| |B| cos(theta)
This tells us that dot product has two parts: Magnitude and Direction
Magnitude means how long or strong the vector is.
Direction means where it points.
The angle tells us whether the two vectors are pointing in a similar direction.
So geometrically, dot product is not just “similarity.”
It is more like:
directional similarity multiplied by strength.
This distinction matters.
Two vectors may point in the same direction, but if one vector is tiny, the dot product may still be small.
Two vectors may be large, but if they are perpendicular, the dot product is zero.
So the geometric view tells us:
Dot product is large when vectors are both strong and aligned.
That is a very useful mental model.
7. How are the algebraic and geometric intuitions related?
The algebraic view says:
Add up agreement across dimensions.
The geometric view says:
Measure alignment in space.
These sound different, but they are deeply connected.
Because “direction” in high-dimensional space is created by the pattern of values across dimensions.
If two vectors have similar coordinate patterns, they point in a similar direction.
For example:
A = (2, 4, 6)
B = (1, 2, 3)
These vectors point in the same direction because B is just a scaled-down version of A.
Their coordinate patterns agree perfectly.
Algebraically, their dot product is high.
Geometrically, their angle is zero.
So the relationship is:
Coordinate-wise agreement creates directional alignment.
That is the key.
The algebraic formula is how we calculate it.
The geometric formula is how we understand it.
This is why dot product is so powerful: it lets a machine calculate something mechanical, while the result carries geometric meaning.
8. Why dot product matters so much in transformers
Now let’s connect this to AI.
A transformer does not see words the way we see words.
It sees tokens as vectors.
For example, the sentence:
“The dog chased the ball because it was excited.”
The model has to figure out what “it” refers to.
Is “it” the dog?
Is “it” the ball?
To answer that, the model needs a way to compare tokens with each other.
This is where attention comes in.
The core question in attention is:
For this token, which other tokens should I pay attention to?
Dot product helps answer that question.
I’ll cover the details of self-attention mechanism in another article as this one was mostly to understand the intuition behind the dot product and why it’s importnat for the transformer models.
9. Bringing it all together
The dot product can be seen in two ways.
The algebraic way:
Compare matching dimensions and add up the agreement.
The geometric way:
Measure how strongly two vectors point in the same direction.
The derivation shows that these are not separate ideas.
They are the same idea.
Coordinate agreement creates directional alignment.
And directional alignment is exactly what transformers need.
Because when a transformer processes text, it is constantly asking:
Which tokens are aligned with what this token is looking for?
That is attention.
And underneath attention is the dot product.
That is why the dot product matters.
It is one of the tiny mathematical gears that makes modern AI feel intelligent.
