Understanding Attention

The attention mechanism is an important component of almost all the AI models being used today.

In 2017, the Transformer model [1] was introduced, making use of self-attention for the first time as the sole mechanism in a encoder decoder architecture. Since then, AI companies have used aspects of this Transformer model in their own systems.

Before we talk about what attention is and how it works, we must go back to 2014.

At that time, Neural Machine Translation was a popular point of research, and many encoder-decoder architectures were invented to tackle this problem. Many of these architectures made use of Recurrent Neural Networks (RNNs).

Very basically, RNNs [2] were models which had a hidden state, and as new inputs were processed, their hidden states were used in transforming the input into an output.

Figure 1. Feedforward and recurrent neural networks. Source: Robin M. Schmidt (2019), Recurrent Neural Networks (RNNs): A gentle Introduction and Overview, Figure 1, p. 2.

The current hidden state was gotten from a function of the input and previous hidden state hₜ = f(xₜ, hₜ₋₁) and the hidden state was updated at every position. But RNNs had an issue. People noticed that as the input sequences got longer, the performance of the RNN deteriorated.