Transformer Block
It is a block with a mechanism to generate contextualized embeddings from static embeddings. It is based on the Transformer architecture (2017, Vaswani A. et al).
Remainder that the text is converted into tokens with their IDs, and these are then converted into static embeddings, which do not include information about the context they currently belong to. If we have:
“Alice spoke to the cat”
and
“The cat spoke to Alice”
Embeddings for tokens common in both texts, such as “Alice”, will return the same embeddings in both cases despite differing contexts. Contextualized embeddings solve this problem.
Architecture
The complete Transformer architecture is larger than what is shown here. Here I show a basic and general representation of a Transformer block:
Self-Attention
This is the general formula for Attention. It uses three matrices: Q , K, and V, whose initials come from “ Query ”, “ Key ”, and “ Value ”, respectively:
These matrices are the result of multiplying the input matrix X (which contains all the input embeddings) with three matrices with pre-trained weights (W) associated with Q , K and V :
Once Q , K and V are calculated, in the Attention formula, Q is multiplied by the transpose of K:
This results in a square matrix, called the Attention Score, which contains the values of how much one token “attends” to other tokens within the input. Both rows and columns represents the input tokens, and the intersection between them stores de attention score corresponding to each pair of tokens. The higher the attention score, the more a token is relevant to the other token.
Mathematically, the higher the attention score, the closer these vectors are to each other in the vector space (i.e., they are more related in the context in which they are).
The next step is division by the square root of the dimensionality of the vectors in K…
To understand the reason for this division, we must first understand the next step in our formula: the softmax .
Softmax is a function that converts a vector of values into a probability distribution: a vector with numbers representing the percentage they represent within the vector. It uses an exponential distribution (with the number e) and the sum of all the values in the resulting vector is always ~1.0 (i.e., 100%).
One problem with softmax is that it can assign a higher percentage to certain values if the range of values within the vector is very large, or it can assign practically zero to very small values. This division scales the Attention Score values, resulting in a better probability distribution among the values.
After the division, we go through the softmax, whose function was explained in the previous step.
The result is then multiplied by the matrix V (it is “projected” onto V , in the mathematical slang), adding information provided by each input token .
Get Alex’s stories in your inbox
Join Medium for free to get updates from this writer.
In summary: Query, Key and Value would represent:
- Query and Key: how much each input token attend to all input tokens (including themselves).
- Value: what information each input token provides.
Add & Norm 1
The output of the Self-Attention block is passed as input to an Add & Norm block, which adds that output to the original input matrix and then applies a normalization function to the result. The goal of each step is to preserve information and stabilize the training, respectively.
In the next block, this output becomes the input for a small feed-forward neural network : it typically has two layers, with an activation function between layers, making it Layer 1 → Activation function → Layer 2 .
The goal is to refine the result through nonlinear transformation, allowing for more complex representations. As in any neural network, the W’s are matrices with pre-trained weights and the b’s (bias) are also pre-trained, making our input (Z) the only variable.
Add & Norm 2
The result of the neural network goes through a second Add & Norm for the same reasons as in the first case, giving as a final output the contextualized embeddings.
Stack of Transformers and Heads
So far I presented a single transformer block; in practice, models use several stacked transformer blocks, where the output of one transformer becomes the input of the next transformer, resulting in more refined contextualized embeddings after each block.
Similarly, in practice, each transformer block has several Self-Attention blocks, each with its own pre-trained matrices (W) for Q , K, and V, which are calculated in parallel with their own Attention formula, and whose results are concatenated into a single final result. These Self-Attention blocks are called heads, and these heads are under a single block called Multi-head Self-Attention (in place of the Self-Attention block).
Positional Encoding
An important part of the transformer is positional encoding. Self-Attention alone doesn’t know the position of the tokens within the text, so “Alice” in “Alice spoke to the cat” will be the same to the Self-Attention as “Alice” in “The cat spoke to Alice”. This is due to a property called permutation equivariance, where changing the order of the tokens in the text produces the same Attention Score.
This is solved by adding positional information (using positional encoding), which makes the same embedding look different depending on the position of its token in the text. There are two approaches to positional encoding:
Absolute Positional Encoding
The encoding algorithm is applied before the input embeddings enter the first transformer of the model. It is basically a vector addition operation, where the input embedding vectors are added to one or more vectors containing positional information (these vectors are called positional embeddings).
Depending on the algorithm, positional embeddings are either pre-trained or algorithmically generated using some formula.
Relative Positional Embedding
Here the Attention formula is modified within the Self-Attention block of each transformer block, and this modification is what adds the positional information.
Summary
- The transformer block provides a mechanism for generating contextualized embeddings from input embeddings (which can be static or contextualized).
- The typical transformer block goes through: Input Embeddings → Self-Attention → Add & Norm → Feed-Forward Neural Network → Add & Norm → Contextualized Embeddings. Positional encoding is also applied , either before entering the transformer block or during Self-Attention , depending on the algorithm used.
- Self-Attention calculates how relevant each token is to all tokens, including itself.
- Add & Norm (in both blocks) are for preserving information and stabilizing training.
- The feed-forward neural network is for refining representations, allowing for more complex representations.
- In practice, the models use several stacked transformer blocks, each with many individual Self-Attention blocks (called heads ), processed in parallel and whose results are concatenated as a single result.
- Positional encoding is an algorithm that allows Self-Attention to consider the position of tokens in the input text. This can be done before the first transformer (absolute PE) or as part of Self-Attention within each transformer block (relative PE).
Sources
AI assistants consulted:
- ChatGPT
- Perplexity
YouTube channels consulted:
Formulas generated by ChatGPT and Perplexity.
Images created by me.
