Encoder-Decoder
To deal with the Seq2Seq problems we have, Encoder-Decoder models:
Overview:
The architecture consists of 2 seperate blocks, which is Encoder and Decoder. These are connected with a context vector.
(i) Encoder :
Take the input sequence as input, the input sequence is tokenized before passing. The encoder will then summarise the input statement and produce a context vector.
(ii) Decoder :
Takes the input as the context vector and returns tokens sequence as output , that is then converted to the language.
Here as we can see, in case of machine translation, the decoder is working or generating output using predict next word.
Training(for Machine Translation-E.g)
Dataset used for Training:
These datasets are known as parallel datasets. It looks like 2 columns, that have the source language word and it’s corresponding meaning in desired language.
E.g :
Let’s consider a very small dataset, just for understanding how the encoder and decoder works.
An E.g Dataset that we will use :
Using this we will see step by step how the training works simultaneously for Encoder and Decoder.
1.) Tokenization :
These natural language words will be first converted to tokens. using OHE, we can convert each word to token in our dataset.
One Hot Encoding :
The sentences will be converted into tokens.
Let’s consider in our training, the 1st row:
Row 1 — — → [ Think about it — — → सोच लो ]
One thing to note is that, the Encoder & Decoder will be trained simultaneously. The weights will be initialized randomly for the encoder and decoder.
So, Let’s see the encoder and decoder part side by side
For Row 1 — — → [ Think about it — — → सोच लो ]
Let’s take the Encoder:
So, here for the encoder,
at Time Step : 1, the word “Think” will be given to the encoder which has nothing but LSTM architecture.
So, the LSTM will have C(t) , h(t) from previous process,
Now, it will take “Think” at Time step 1 and it will update it’s hidden state plus the current state.
This way we will pass “About” in Time step 2 and subsequently the word “it” at Time step 3. So this is how the encoder will work and at the end we will have some updated C(t) and h(t). Using the final C(t) and h(t) we will form our context vector.
This context vector will be sent to the Decoder.
So, for t = 1, It will give us an output, which we will run through the Softmax layer, to get the probabilities for the predicted word.
Now here, according to the diagram, that word is “ सोच ”.
Now, we are in prediction phase and not in training, so we are not going to do any kind of “Teacher Forcing”, whatever prediction we have at Time step = 1, we will send that to Time step = 2 as input. So If “ सोच ” is predicted as output of Time step = 1, then it will be the input for Time step = 2
This is how next word prediction works.
So, for t = 2, the input will be “ सोच ” and the C(t) , h(t) values coming from t = 1, whatever it predicts : here it is “ जाओ ” , It will be input for t = 3.
For t = 3, the input is “ जाओ ”, the output is “ लो ”
Again at t = 4, the input is “ लो ”, output is “ <end> ”
The moment <end> is seen the process of prediction/next word prediction stops.
Note: The next word prediction will go on until the predicted word turns out to be <end>.
Finally:
Input : [ Think about it ]
Output:[ सोच जाओ लो]
2.) Prediction:
Now, like this the training is done all the data and a final Weight & Bias is updated.
So, now we see how the prediction is working here:
At Encoder:
Let’s talk about the encoder part first, The encoder part remains pretty much the same, we can see for encoder , at Time step = 1
We send the OHE value of input i.e., “Think” , The LSTM updates it’s C(t) and h(t), eventually at Time step 2 : “ about ” is sent as input and we have the updated C(t) and h(t) coming from t=1.
Get Rahul’s stories in your inbox
Join Medium for free to get updates from this writer.
Again same process happens for the third time step = 3 where input is “ it ”. Now we have finally the h(t) and C(t) values from the encoder.
These values are the Context Vector , which is ready to be sent to the decoder.
At Decoder:
At Time step = 1 , we know the input will be “ <start> ” and we have the h(t) and C(t) values portrayed as context vector coming from the Encoder.
So, how this simultaneous training looks at the Decoder side, let’s look at it:
The context vector, which was given to us by the encoder is given as the input for Time step — 1, where the input to LSTM of Decoder is <start>.
The LSTM uses the previous context, in this case the “ h(t-1) ” and “ C(t-1) ” for this LSTM, and uses the input at Time step — 1 i.e., “ <start> ”.
This gives an intermediate output at Time step = 1, this output is sent to a softmax layer, this Softmax layer is attached just after the output of the LSTM at Time step =1.
This Softmax layer has the same number of nodes as the length of the vocabulary used for decoder or the number of words used in the vocab of Decoder.
The decoder part is having a total of 7 words, in its vocabulary.
Each node will output a probability for that corresponding word, and whichever node value is higher, it is the word as output
For E.g : The Softmax layer outputs something like this for our Time step =1
The highest probability is for the 3rd index of that list
So, that will be our output, If we map the third word to our vocab.
Then it is mapped to the word “ लो ”
So, the output will be the O.H.E of the word “ लो ” which is
[ 0 , 0 , 1 , 0 , 0 , 0 , 0 ]
Now, Here’s a catch,
According to our words, the output for this Time step = 1 is “ सोच ” and not “ लो ”. But as our model is being trained for the first time this error is expected where our:
y_true = [ 0 , 1 , 0 , 0 , 0 , 0 , 0 ]
y_pred = [ 0.01 , 0.1 , 0.4 , 0.02 , 0.002 , 0.02 , 0.07 ]
So, clearly :
y_true = “ सोच ” but,
y_pred = “ लो ”
But according to the concept of next word predictor or the concept that we are going to use for Decoder part of training:
The predicted word at Time step = 1 will be used as the input for the next Time step , i.e., t = 2.
But our decoder has predicted a wrong word at t = 1,
So, should we send the wrongly predicted word as input for the next Time step, t = 2 or not?
Answer is : We can send this wrongly predicted word, that is not a problem, but it will slow down the training process,
That’s why we use : Teacher Forcing
Sending the predicted word to Time step = 2, will not be a problem, but there are good amount of chances that the predicted word is wrong, and if we send the wrongly predicted word to Time step = 2, then the training process will be very slow.
So, Instead of sending the predicted word of Time step = 1 as input to the Time step = 2. We will be sending the correctly mapped word at Time step = 2 in input — This is called Teacher Forcing
So, whatever maybe the predicted result at Time step = 1, Here we had the prediction for t = 1 as “ लो ”, but we wanted “ सोच ”,
So that we can feed the correct input to Time step = 2
So, we simply ignore the output/prediction of time_step = 1 and force/manually put “ सोच ” as input to Time step = 2
This is what is Teacher Forcing: it looks like:
So, now let’s continue our way to Time step = 2, here we are feeding the h(t) and C(t) from Time step = 1 and we are providing a correct input : “ सोच ”
Here we can see, that the output after being passed through the softmax layer gives probability, and according to that score our prediction seems to be : “ जाओ ” which is again not true, So we will again do “Teacher Forcing” for Time step = 3.
Now for Time step = 3
we will feed the correct input to the LSTM i.e., “ लो ” with the values of C(t) and h(t) from Time step = 2,
When we calculate it for
Time step t = 1 ==> L(1) = -1log(0.1) = 1
similarly for
Time step t = 2 ==> L(2) = -1log(0.1) = 1
Time step t = 3 ==> L(3) = -1log(0.4) = 0.39
Overall loss = 1 + 1 + 0.39 = 2.39
Average loss = 2.39/3 = 0.79
Note : For the first 2 time steps the prediction was wrong, So the loss is extremely high.
Now we have done the Forward Propagation, Loss computation
Time to explore the Backward Propagation
Back propagation has 2 parts : (i) Gradients and (ii) Weight updation
Back propagation works here the same way, it did in case of RNN i.e., through BPTT
So, overall flow becomes something like:
Here we can see, the output after the Softmax layer shows that the predicted word is “ <end> ”, which is true
Note : The prediction process will go on till <end> is not emitted as an output, Here to make the process simple we assumed at t=3 the output would be “ <end> ” only, but if it would have been something else then we would have had a further Time step t = 4 , and so on…. until at one of the Time step “ <end> ” was emitted as output. So we stop the next word prediction when we encounter <end> as output.
Now, with this we are done with the Forward propogation of our training
It’s time to compute the loss:
Loss:
For the 3 Time steps of the Decoder we have the loss
Thank you Very much for following up,
Up next we have the Attention Mechanism.




![RLHF Series: 15/17 [Regularization in RLHF ]](https://miro.medium.com/v2/resize:fit:1358/format:webp/1*k_vYiZ88MSd5nIAuogLYMQ.png)




