Sitemap
saahil

saahil

Data scientist/engineer (current). PhD in software engineering, security testing, deep learning, effective vulnerability management. http://saahil.github.io

Trouble With Training RNNs

Follow up post on Recurrent Neural Networks

4 min readJun 7, 2014

I introduced RNNs in an old post in a very superficial manner. Recurrent networks are semantically applicable to tasks that are serial in nature or follow temporal consistency, e.g. predicting natural language, stock market trends, music progressions etc.

Let’s revisit these networks and try to see why they are difficult to train with conventional gradient descent methods.

Schematic of RNN. (On the difficulty of training Recurrent Neural Networks by Pascanu et al.)

In the schematic of an RNN, the hidden states depend on the preceding state values by a recurrent weight matrix, W_rec.

(I’m going to use W in place of W_rec in the rest of the essay, because Medium doesn’t let me subscript. Input to hidden weights, W_in, plays the usual role as in an MLP)

Another popular variant of equation (2) is one where the non-linearity, σ, is applied on the linear combination of all weights and biases. But we will continue with this definition for the sake of convenience.

Recall the “unrolled” architecture of an RNN, where individual time steps’ MLPs are fully connected with the input and output nodes, and the hidden layers are fully connected to adjacent time units.

Press enter or click to view image in full size
Taken from Ilya Sutskever’s thesis

If we train such a network using backpropagation algorithm, where the gradient of loss at a respective node output is used to descend the weight values, so as to reach a minima in the error function, the recurrent matrix, W, will be compounded per time step. This would be harmful as far as storing long term time dependencies goes, because by the time the error gradient at t would have been propagated to t’<<t, the norm compounding in W would have overpowered the time-dependency effect.

Get saahil’s stories in your inbox

Join Medium for free to get updates from this writer.

The compounding of the error gradient can happen in one of two opposite directions, both depending on the largest eigenvalue (also known as the spectral-radius) of the recurrent weights matrix. If the spectral radius is much less than 1, the gradient would vanish over time. On the other hand, if the spectral radius is bigger than 1, the gradient would explode over time.

Let’s demonstrate the delicate nature of training recurrent weight matrix, using an over-simplified architecture.

In (2), assume that there is no new input coming at every time step, so that the second term with u_t becomes unnecessary. Furthermore, assume that x is a single dimension variable, which means that W and b have shapes [1, 1] and [1] respectively.

Our objective, then, is to start x with a zero value and reach a given target value, z in a set number of time steps. I chose z as 0.78 and 50 time steps, for no reason in particular. I present the training graph over 10000 different initialization sets of W and b.

Press enter or click to view image in full size

The steep wall perpendicular to the parameter space represents an explosion in gradients of the loss function. Basically, when the largest eigenvalue vector in the parameter matrix is compounded, the curvature of the error surface compounds too and that’s what the wall illustrates.

But what is noteworthy is this — When the search routine is at a point on the top surface of the error curve, it makes it’s next step in a direction perpendicular to the face of the wall. Depending on the learning rate, it might then fall to ground “beyond” the valley where the error reaches it’s minimum. This is not such a big problem, because the search must come back to the valley region, given itself to explore the ground region.

Unless! Unless of course the search direction bangs head-on into the wall again, at which point a small change in the norm of the update would take the search back to the top of the hill.

Press enter or click to view image in full size
A more desirable training regime. Note the use of ‘J’ instead of ‘L’. This is because the raw error ‘L’ has to be regularized somehow to make the search space smoother.

Norm-based regularizers and gradient clipping methods might solve the problem of exploding or vanishing gradients to some extent, but they depend heavily on initialization of parameters in a right regime.

In a few weeks’ time, I’ll describe some analytical and structural methods to deal with these training issues, which have been known to work well.

If you enjoyed this article, please recommend, tweet or share using the buttons below. That would be ace! ☺

--

--

saahil
saahil

Written by saahil

Data scientist/engineer (current). PhD in software engineering, security testing, deep learning, effective vulnerability management. http://saahil.github.io