Understanding LLMs

Chapter 2.1: Understanding Sinusoidal Positional Encodings

Note : This blog is a continuation of our chapter 1, where we discussed about self and multi-head attention. I would suggest to have a thorough understanding on that before one comes to understand positional encodings. You can read that blog here:
https://medium.com/@aditya.p22/understanding-llms-9e5486ce36a9

Introduction — The need for positional encodings

When it comes to teaching machines how to understand human language, one of the most subtle yet powerful signals is word position. The order of words in a sentence isn’t just a matter of grammar — it’s often the key to meaning.

Take a look at these two sentences:
“I have an apple every single day”
“Apple’s latest software update isn’t cool”

The word “Apple” appears in both, but in vastly different contexts. In the first, it’s a fruit. In the second, it’s the tech giant. So, how does a model learn that?

That’s where positional information comes in.

Traditional word embeddings (like Word2Vec or GloVe) treat each word independently. They don’t capture the order of words in a sentence. But in natural language, position can change everything.

To make models aware of word order, we need a way to inject positional context into the model. In other words: not just what the word is, but where it is in the sentence.

Enter: Positional Encodings.

Over the years, researchers have proposed various methods to integrate positional information into models. Broadly speaking, these can be divided into two categories:
Sinusoidal Positional Encodings
Rotary Positional Encodings (RoPE)

When “Attention is All You Need” had introduced the Transformer architecture, it brought a revolution in natural language processing. Unlike RNNs, Transformers don’t process input sequentially. They look at all words at once — using self-attention to understand relationships between words.

But this came with a challenge: without processing inputs in order, how does the model know the position of each word? The solution proposed in the paper — Sinusoidal Positional Encodings.

Today’s post is all about understanding Sinusoidal Positional Encodings — why they were needed, how they work, and what advantages they bring to the table.

We’ll also take a look at some of their limitations, especially as models scale and become more powerful.

And in the next blog, we’ll dive into Rotary Positional Encodings (RoPE) — a newer technique that tackles some of the shortcomings of sinusoidal encodings, and powers many of today’s most advanced models.

Building an Intuition : Integer & Binary Positional Encoding

Let us build out intuition about positional encodings in a simple way. Consider this sentence:
“Apple’s latest software update isn’t cool”
For now, we will assume that the sentence is split at a word level. Therefore, a token is a word. I will use the word ‘token’/ ‘word’ interchangeably.

An image showing word level tokenization

As we discussed earlier, traditional word embeddings (like Word2Vec or GloVe) give us a fixed-size vector for each word. These embeddings capture semantic meaning — words that are similar in meaning will have similar vectors.

Let’s assume each word is represented using a 300-dimensional vector. So the sentence gets converted into a 2D matrix where each row corresponds to a word’s embedding:

2-D embedding matrix

The simplest and most intuitive way to add positional information to a word embedding is this:

Just add the position index to the word’s embedding vector.

Let’s say the word “Apple’s” is at position 0, “latest” is at 1, and so on.
Now imagine each token has a 4-dimensional embedding (just to keep things simple):

A 4-d vector (for our understanding)

Now, to this, we will add a 4-d vector, that contains the position of the token.

Positional Encoding for Apple’s

Positional Encoding for latest

You might be wondering:

“Why do we go through the trouble of generating a whole new positional vector and adding it to the embedding? Why not just append the position as a single number at the end of the embedding? Just make the vector one dimension longer?”

Transformer layers, especially self-attention, expect all input embeddings to have the same fixed dimensionality.

If we append a single number to each embedding vector (making it 301-dimensional instead of 300), then the positional info sits in a tiny corner of the input — it’s just one dimension out of 301. The rest of the attention mechanism is mostly blind to it.

In contrast, when we add a positional vector (of the same dimension), the positional information is distributed across all dimensions. This means:
Every neuron, every attention head can “see” position — not just the last dimension.

This type of Positional Encoding, that we just saw, is called “Integer Positional Encoding”

Issues with Integer Positional Encoding:
1. No sense of relative distance — The model just sees “1”, “2”, “3” — but it doesn’t understand that the distance between positions 3 and 4 is the same as between 4 and 5.
2. No generalization to longer sequences — We saw the example of the positional encoding values to be smaller (1 or 2), but in long sequences, these values might increase (200 or 300). Now this causes an issue, because the token embeddings (the semantic meaning) values are small, compared to these values, which means the magnitude of the positional values becomes much larger than the magnitude of the embedding values.

An explanatory image for point 2

So this means that, our semantic meaning is getting polluted. The positional values dominate the entire vector. Instead of the model understanding “this is the word update at position 250,” it only sees “250!”

So this means, we want the positional encodings to be smaller. And this is what binary positional encoding does.

Binary Positional Encoding is based on a straightforward principle:
Take the integer position (e.g., 1, 2, 3, …, 300),
Convert it into its binary representation,
And then use that as your positional vector.

Consider a word at position 5, binary 101.

An example for binary positional encoding.

Here is an important observation we can make:
- The LSB (Least significant bit) oscillates between 0 and 1, and changes in each digit.
- The next bit oscillates between 0 and 1, and changes in every 2 digits.
- The next bit changes in every 4.

This means that the lower indices will oscillate fast, and the higher indices will oscillate slow.
Faster oscillations means faster changes are captured, this simply means that the small distance dependencies will be captured here.
Slower oscillations means that slower changes are capture, and that would mean, the long distance dependencies will be captured here.

If we try to plot these oscillations, it will give a plot in the below format:

Plot of oscillations — topmost is the LSB, and bottommost is the MSB.

So switching to binary encoding helped us by:
1. Making the values of the positional encoding vector smaller (its either 0 or 1).
2. Capturing the relative information (short & long term dependencies) with the help of oscillations.

Get Aditya Pande’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

But, this method has one flaw — It is discrete.
This would cause a problem during back propagation, as the values are not smooth and continuous, making the training unstable.
So, what we need is a function that is:
1. Continuous
2. Bounded
3. Oscillates and repeats, or in simpler terms is periodic.

A tip for visualization purpose, If the plot we saw would have been continuous, it would look very similar to a sin or a cos plot.

Sinusoidal Positional Encoding

The below image tries to summarize what we are trying to achieve in sinusoidal positional encodings:

The smoothening of the plot we had got for binary positional encoding

For a token at position pos, its positional encoding will be a vector of length d — where d is the model’s embedding dimension. Each entry in that vector is computed using a sine or cosine function, depending on whether the index is even (2i) or odd(2i+1).

The positional encoding formula proposed in the attention is all you need paper

I will use f​ to denote 1/10000^(index/dmodel)

Let us first try to understand why sin and cos are considered here, among all the periodic functions that we have.

https://huggingface.co/blog/designing-positional-encoding This huggingface blog, provides us 4–5 points, that we would want our periodic function to follow, to make it “desirable” for positional encoding generation. In this a very important property specified is:

Property 2 — Linear relation between two encoded positions:
If we know the positional encoding at position p, we can derive the positional encoding at position p + k — without recalculating from scratch.

Recall this trigonometric formula we had studied our school time:

sum identities

Now, below is attached a screenshot that will explain to you, how we can use that formula, to satisfy the above mentioned property.

Mathematical derivation of how the property is satisfied

Notice that the components of PE(pos+k,index) can be expressed as a linear combination of the components of PE(pos,index)​, hence statisfying the property. Another thing we can say is that in PE(pos+k,index)we have sin(θ+ω) and cos(θ+ω) , where ω=k*f & θ=pos*f.

Secondly, let us try and visualize what is happening:

Each PE is a vector that has 2 components:
- The X component
- The Y component

This visualization tells us, that we have to just rotate the positional encoding vector at position ‘p’, to get the positional encoding vector of the position ‘p+k’. This would mean that the relative positional encodings is just rotations of each other, and we can cover that using this technique too!

So the primary reasoning of using sin & cos (2 different functions) for different indexes is because:
1. They are continuous, periodic & bounded.
2. They enable relative information capturing, as they provide the components of the next ‘p+k’ positional encodings.

When I first encountered sinusoidal positional encoding, a few interesting questions popped into my head — “The paper arbitrarily chooses sin for even indices and cos for odd ones. Is that necessary? What if I swap them — use cos for even and sin for odd?”

If we swap:
PE(pos,2i)=cos(pos*f​)
PE(pos,2i+1)=sin(pos*f​)

When we swap sine and cosine like this, we are still preserving the core structure:

• We have both sine and cosine components per frequency.
• The trigonometric identities still hold (like sin(a + b) and cos(a + b)).
• The values are still bounded, smooth, and periodic.

What changes is just the phase of the wave — meaning the starting point of the oscillation. This means that we are rotating our coordinate system slightly, but the relative distances, patterns, and periodicity stay the same.

The Transformer model doesn’t “expect” sin or cos at any specific index. It just learns to use whatever pattern is provided to capture positional context. Since a sine and cosine pair is just a rotated version of the same wave, the model can easily adapt — learning from the shifted phase without any loss of meaning.

Another question I remember asking myself — why does it have to be this complicated? Why not just use something simpler, like multiplying the position with the dimension index? Couldn’t we do something like pos * 2i instead?

The core idea behind the sinusoidal formula is this:
We want each dimension in the positional encoding to represent position at a different frequency.

• Some dimensions should capture local, high-frequency variations (small shifts between nearby tokens).
• Others should capture global, low-frequency variations (positions far apart in the sequence).

This allows the model to recognize patterns across both short and long distances. In this proposed formula, for low values of i, the denominator is small, so the sine and cosine functions oscillate quickly — perfect for encoding detailed, local positional differences. As i increases, the denominator grows exponentially, making the frequency lower and wavelength longer — ideal for capturing broad, long-range patterns in a sequence. The below image shows it with an example:

Image showing the “different frequencies” covered

This exponential scaling gives us a smooth spectrum of positional signals, distributed evenly across all embedding dimensions — from the most local to the most global.

The number 10000 might seem arbitrary — and it kind of is. It was chosen empirically by the authors of the original Transformer paper after some experimentation. But it’s not sacred.

You can absolutely try different values, like 1000, 100, or even 10. Changing this base will affect the range and density of the frequencies. Smaller values will produce faster-changing sinusoids, which might be useful for shorter sequences or tasks that benefit from very fine-grained position signals. On the flip side, larger values will spread out the waves even more, prioritizing long-range relationships. That said, 10000 has been found to work well in practice for most standard NLP tasks — it's a solid default.

Wrapping Up: A Solid Step Forward, But Not the End

So it seems that we’ve addressed several important challenges in modeling position:

• Our positional encodings are now continuous and bounded,
• Their values stay small, so they don’t overpower token embeddings,
• And thanks to sine and cosine, they oscillate at multiple frequencies, allowing the model to capture both short-range and long-range dependencies.

It’s a clever, elegant solution — and one that powered the original Transformer architecture into the state-of-the-art spotlight.

But… we’re not done yet.

There’s still one important issue left on the table.

When we add positional encodings to token embeddings right at the start, we’re effectively mixing position with meaning — and potentially polluting the semantic information of the original token.

This may not always be a problem, but in tasks where fine-grained meaning matters deeply — or where we need better relative positioning — this method might blur the lines a bit too early.

And that’s exactly where Rotary Positional Encoding (RoPE) comes in.

Rather than adding position to embeddings at the beginning, RoPE takes a completely different route: it bakes position directly into the attention mechanism itself, preserving semantics while enabling powerful, relative-aware reasoning.

Coming Up Next…

In the next blog, we’ll dive deep into Rotary Positional Encoding — how it works and how it solves some of the subtle limitations of sinusoidal encodings in an incredibly elegant way.

Thank you for reading this blog and I hope you have a great day!
Stay tuned! 🔥