Deep Learning-Simple RNN
Drawbacks of ANN :
In text data sequence of information is important and ANN does not maintain the sequence and hence meaning/context is lost. For sequential of information, data should be provided one by one but in ANN all the row of data is passed at once
Simple RNN :
RNN means recurring neural network and the most important feature is it maintains sequence and hence context is not lost.
In RNN , we will have feedback loop which will pass the output of hidden neuron to same neuron and also the other neurons in same hidden layer
In RNN we pass input at each timestamp
Eg : t1 = x11 , t2 = x12, t3 = t13 etc …
When we pass first word at t1 it will be available in all hidden neurons on 1st hidden layer due to feedback loop and when we pass second word at t2 then the same neuron will first word as well as second word and hence sequence is maintained which helps in attainig the context.
Below is the diagram when we UNFOLD a neuron in hidden layer
Forward propagation with timestamp:
Dataset as below
At time t1 when we pass through hidden layer, each input will have some weights W assigned and we will get output as O1, then feedback loop will be applied and during this process W1 weight be assigned to neurons in hidden layer. At timestamp t1 we will have O1, t2 : O2, t3:O3 and final output as O4.
Weight calculation formula is as below. For O2,O3 we will have normal weights along with the weights after feedback loop which helps in maintaining the context
Weights wont get updated during forward propagation and this will be get updated during backward propagation
Backward propagation with timestamp:
In backward propagation below is the order
Get Rajasekhar’s stories in your inbox
Join Medium for free to get updates from this writer.
Loss is dependent on predicated output → Predicated output is dependent on O3 → O3 is dependent on O2 → O2 is dependent on O1
We will have three types of weights in RNN
- Wi → Weights of input
- Wh → Weights of hidden layer
- W0 → Weights of output
The general formula for weight updation is as below and in RNN weight updation happens w.r.t to timestamp
In Backward propagation
Step1 :we first need to update the output weights W0 . For calculating derivative (to know SLOPE) we will use the chain rule and formula is given as below for each timestamp
Step2: We need to update the hidden layer weights Wh at each timestamp t1,t2 and t3 and the formulae is given as below.
In t3 we will context of last word, t2 context of third and second word and at t3 we will have first,second and third word context
Step3: We need to update the input weights Wi at each timestamp t1,t2 and t3 and the formulae is given as below.
Our end goal is to bring all the weights towards global minima to reduce loss
Problem with RNN :
It will have vanishing gradient descent problem.
Suppose if we have 100 words then we will t-100 , the output word may dependent on word that started earlier and this dependency cannot be captured by simple RNN.
Long tern dependency cannot be captured by RNN and hence it cannot provide the better accuracy due to derivative value lies between 0 to 0.25 and hence at some point the updated weights will be very small and hence convergence towards global minima does not happen
To fix this issue can use LSTM RNN ( Long short term RNN ) or GRU RNN
