The Complete Guide to Recurrent Neural Networks

Standard RNN’s challenges

Training a RNN or be it any Neural Network is done by defining a loss function that measures the error/deviation between the predicted value and the ground truth. The input features are passed through multiple hidden layers consisting of different/same activation functions and the output is predicted. The total loss function is computed and this marks the forward pass finished. The second part of the training is the backward pass where the various derivatives are calculated. This training becomes all the more complex in Recurrent Neural Networks processing sequential time-sequence data as the model backpropagate the gradients through all the hidden layers and also through time. Hence, in each time step it has to sum up all the previous contributions until the current timestamp.

Exploding gradients

In some cases the value of the gradients keep on getting larger and becomes infinity exponentially fast causing very large weight updates and gradient descent to diverge making the training process very unstable. This problem is called the exploding gradient.

Vanishing gradients

In some other cases, as the background propagation advances from the output layer to the input layer, the gradient term goes to zero exponentially fast, which which eventually leaves the weights of the initial or lower layers nearly unchange and makes it difficult to learn some long period dependencies.  As a result, the gradient descent never converges to the optimum. This problem is called the vanishing gradient.

Recurrent Neural Networks Architectures

Bidirectional recurrent neural networks (BRNN)

Bidirectional Recurrent Neural Network

A typical RNN relies on past and present events. However, there can be situations where a prediction depends on past, present, and future events.

For example, predicting a word to be included in a sentence might require us to look into the future, i.e., a word in a sentence could depend on a future event. Such linguistic dependencies are customary in several text prediction tasks. 

Thus, capturing and analyzing both past and future events is helpful.

To enable straight (past) and reverse traversal of input (future), Bidirectional RNNs or BRNNs are used. A BRNN is a combination of two RNNs - one RNN moves forward, beginning from the start of the data sequence, and the other, moves backward, beginning from the end of the data sequence. The outputs of the two RNNs are usually concatenated at each time step, though there are other options, e.g. summation. The individual network blocks in a BRNN can either be a traditional RNN, GRU, or LSTM depending upon the use-case.

Gated Recurrent Units (GRU)

There can be scenarios where learning from the immediately preceding data in a sequence is insufficient. Consider a case where you are trying to predict a sentence from another sentence that was introduced a while back in a book or article. In this case, remembering the immediately preceding data and the earlier ones is crucial. A RNN, owing to the parameter sharing mechanism, uses the same weights at every time step. Thus back propagation makes the gradient either explodes or vanishes, and the neural network doesn’t learn much from the data, which is far from the current position.

GRU uses update and reset gate. Basically, these are two vectors which decide what information should be passed to the output. The special thing about them is that they can be trained to keep long-term information without washing it through time or remove information which is irrelevant to the prediction.

• The update gate is responsible for determining the amount of previous information that needs to pass along the next state. 

• The reset gate is used from the model to decide how much of the past information is needed to neglect

Long Short Term Memory (LSTM)

‍

A LSTM is another variant of Recurrent Neural Network that is capable of learning long-term dependencies. Unlike in an RNN, where there’s a simple layer in a network block, an LSTM block does some additional operations. Using input, output, and forget gates, it remembers the crucial information and forgets the unnecessary information that it learns throughout the network.

• Input gate finds which value from input should be used to modify the memory.

• Forget gate learns what details to be discarded from the block.

• Output gate discovers the input and the memory of the block is used to decide the output.

The key difference between GRU and LSTM is that GRU's architecture has two gates that are reset and update while LSTM has three gates that are input, output, forget. GRU is less complex than LSTM because it has less number of gates. Hence, if the dataset is small then GRU is preferred otherwise LSTM for the larger dataset.

Pro-tip: Read this article on AI-Generated Art: From Text to Images that outlines some examples where RNN plays a crucial role.

Key Takeaways

• Recurrent Neural Networks, or RNNs, are a specialized class of neural networks used to process sequential data. 

• Modeling sequential data requires persisting the data learned from the previous instances. RNN learns and remembers this data so as to formulate a decision, and this is dependent on the previous learning.

• It implements Parameter Sharing by using same weights at every time stamp so as to accommodate varying lengths of the sequential data for which it makes use of feedback loops.

• One main limitation of RNN is that the gradient either explodes or vanishes; The network doesn’t learn much from the data which is far away from the current position.

• RNNs have short term memory problem. To overcome this problem specialized versions of RNN are created like LSTM, GRU.

• Another limitation of RNN is that it processes inputs in a strict temporal order. This means current input has context of previous inputs but not the future. 

• Bidirectional RNN (BRNN) duplicates the RNN processing chain so that inputs are processed in both forward and reverse time order.

• RNNs are widely used in the following domains/ applications: Machine Translation, Speech Recognition, Generating Image Descriptions, Video Tagging, Text Summarization etc.