Recurrent Neural Networks (RNN) | TrendSpider Learning Center (2024)

15 mins read

A Recurrent Neural Network (RNN) is a type of artificial neural network where the output of certain layers is stored and fed back into the input. This mechanism helps in predicting sequential data by using past information to inform future predictions. Initially, the first layer processes data similarly to a feedforward network, using the product of weights and features.

However, what sets RNNs apart is their “memory,” which allows them to use information from previous inputs to influence the current input and output. While traditional deep neural networks treat inputs and outputs as independent of each other, the outputs of RNNs depend on the prior elements within the sequence.

Leveraging their memory, RNNs excel at handling sequential data where the order of elements is crucial. This makes them ideal for tasks such as speech recognition (Graves et al., 2013), language modeling (Graves, 2013; Pascanu et al., 2013a), learning word embeddings (Mikolov et al., 2013a), and text generation, where the sequence of words or sounds impacts the overall meaning.

Moreover, as RNNs process new data, they continuously update their internal memory. This dynamic processing capability allows them to adapt to changing patterns within a sequence, making them versatile for various applications that involve evolving data streams.

Another distinguishing characteristic of recurrent neural networks is their parameter-sharing across each layer. Unlike feedforward networks, where each node has different weights, RNNs use the same weight parameters within each layer. These shared weights are adjusted through backpropagation and gradient descent to optimize learning and improve performance.

Recurrent Neural Networks (RNNs) consist of neurons, which are data-processing nodes that collaborate to perform complex tasks. These neurons are organized into three layers: input, hidden, and output. The input layer receives the information to be processed, the hidden layer handles data processing, analysis, and prediction, and the output layer generates the final result.

Further details are discussed in the section- RNN Architecture below.

Source: AWS

Origin & HIstory

From the early work on Hopfield and Elman networks to the development of LSTMs and GRUs, each advancement has contributed to the ability of RNNs to capture and utilize temporal dependencies. Modern techniques like attention mechanisms and transformer models have further expanded the capabilities of neural networks in processing sequential data, solidifying RNNs, and their derivatives as foundational tools in the field of deep learning.

I. Hopfield Networks (1982)

Hopfield Networks, a pioneering form of Recurrent Neural Networks (RNNs), were first introduced by John Hopfield in his 1982 paper titled “Neural Networks and Physical Systems with Emergent Collective Computational Abilities“. These networks represent the earliest example of associative neural networks, capable of producing emergent associative memory. Associative memory, also known as content-addressable memory, allows the system to recall stored memories based on their similarity to an input pattern, enabling robust memory recall through pattern association.

II. Elman Networks (1990)

Elman proposed a simple recurrent network (SRN), later known as the Elman Network in his paper- Finding Structure in Time (Elman, 1990). It was developed to address the need for temporal processing in connectionist models, especially for tasks involving sequences. The following are the components of the Elman Network:

• Input Layer: This layer receives the initial data or sequence. For example, in a language task, the input might be a series of words or letters.
• Hidden Layer: This layer processes the input data and captures patterns. In the Elman Network, the hidden layer not only processes current inputs but also retains information from previous time steps.
• Context Layer: The context layer stores the outputs of the hidden layer from the previous time step and feeds them back into the hidden layer. This acts as a form of memory, allowing the network to maintain information over time and use past inputs to inform future outputs.
• Output Layer: This layer generates the network’s final output based on the current input and the stored context. This might be a prediction or classification relevant to the task at hand.

III. Jordan Networks (1986)

Michael I. Jordan introduced another type of recurrent network, similar to Elman Networks, where the context units were fed from the output layer instead of the hidden layer (Jordan, 1986). These networks involve connections from the output units back to a set of context units, which serve as additional inputs in the next time step. This feedback mechanism allows the network to retain a form of memory from previous outputs, enhancing its ability to process sequences of data. By using context units derived from the output layer, Jordan Networks can capture temporal dependencies and improve performance on tasks involving sequential data.

IV. Backpropagation Through Time (1990)

Backpropagation Through Time (BPTT) is a gradient-based algorithm used for training Recurrent Neural Networks (RNNs), specifically addressing the challenges posed by the temporal dependencies in sequential data. Developed by Paul Werbos in 1990 (Werbos 1990), BPTT extends the conventional backpropagation algorithm used in feedforward neural networks to RNNs by unfolding the network through time. This process involves replicating the network structure for each time step, effectively transforming it into a multi-layer feedforward network, where each layer corresponds to a different time step.

V. Advanced Architectures

Advanced architectures such as Long Short-Term Memory (LSTM) networks, Gated Recurrent Units (GRUs), and Bidirectional Recurrent Neural Networks (BRNNs) have significantly improved the capabilities of traditional RNNs. Introduced by Sepp Hochreiter and Jürgen Schmidhuber in 1997, LSTMs address the vanishing gradient problem with memory cells and gating mechanisms to retain information over long periods, enabling the learning of long-term dependencies. In 2014, Kyunghyun Cho and colleagues developed GRUs as a simpler alternative to LSTMs, merging the cell state and hidden state and using two gates (reset and update) to reduce computational requirements while maintaining performance. Additionally, Mike Schuster and Kuldip K. Paliwal proposed BRNNs in 1997, which enhance context understanding by processing data in both forward and backward directions, utilizing information from both past and future input sequences.

RNN Architecture

In the RNN architecture, data flows through the input layer, is processed in the hidden layer with recurrent connections, and results in outputs at each time step. The hidden states act as memory cells, retaining information across the sequence, while activation functions add non-linearity to the model, enhancing its ability to learn complex dependencies within the data. This structure allows RNNs to effectively handle tasks involving sequential and time-dependent data.

Source: https://stanford.edu/

The diagram above illustrates the architecture of a regular Recurrent Neural Network (RNN), showcasing the flow of information through its various components: input layer, hidden layer, output layer, activation function, and recurrent connection.

I. Input Layer

• Function: This layer receives the initial element of the sequence data. For instance, in a sentence, it might take the first word as a vector representation. After receiving an input, the input layer passes it to the hidden layer.
• Components: The inputs (x<1>, x<2>, …, x<t>) are fed into the network at each time step.
• Role: Each input at a specific time step is processed sequentially, providing the data that influences the hidden states and subsequent outputs.

II. Hidden Layer

• Function: The hidden layer is where the main data processing, analysis, and prediction occur.
• Components: Neurons in the hidden layer (a<0>, a<t-1>, a<t>, a<t+1>) represent the network’s memory, retaining information from previous inputs.
• Role: The hidden states capture the temporal dependencies and contextual information across the sequence of inputs. The hidden state at time t (a<t>) is influenced by the previous hidden state (a<t-1>) and the current input (x<t>).

III. Output Layer

• Function: The output layer generates the final result based on the processed information from the hidden layer.
• Components: Outputs (y<1>, y<2>, …, y<t>) are produced at each time step, representing the predictions or final processed data.
• Role: Each output is derived from the corresponding hidden state, providing a sequential output that correlates with the input sequence.

IV. Activation Function

• Function: Activation functions introduce non-linearity into the network, enabling it to learn complex patterns.
• Common Activation Functions:

~ tanh: Compresses the values between -1 and 1.

~ ReLU: Activates only positive values, setting negative values to zero.

• Role: Applied to the hidden states and sometimes to the output states, activation functions transform the linear combinations of inputs and weights into non-linear outputs, allowing the network to capture more intricate patterns in the data.

V. Recurrent Connection

• Function: The recurrent connection allows information to be passed from one-time step to the next, maintaining the network’s memory.
• Components: The hidden state (a<t>) at each time step (t) is influenced by the previous hidden state (a<t-1>) and the current input (x<t>).
• Role: This connection creates a loop within the hidden layer, enabling the RNN to retain and utilize information from previous time steps, which is crucial for processing sequential data.

RNN Architecture Variants

Each RNN variant offers unique advantages and is suited to different types of sequential data problems. LSTMs and GRUs address long-term dependency issues and BRNNs capture comprehensive context by processing sequences bidirectionally. By selecting the appropriate RNN architecture variant, practitioners can effectively address specific challenges in their sequential data processing tasks.

I. LTSM

Long Short-Term Memory (LSTM) is a variant of Recurrent Neural Networks (RNNs) that extends the memory capacity to accommodate longer time sequences. While traditional RNNs can only remember immediate past inputs, LSTMs can retain information from several previous sequences, enhancing their prediction accuracy.

It is a popular RNN architecture introduced by Sepp Hochreiter and Juergen Schmidhuber (Hochreiter & Schmidhuber, 1997).

LSTM networks incorporate special memory blocks called cells in the hidden layer. Each cell is regulated by an input gate, output gate, and forget gate, allowing the network to retain useful information.

• Memory Cells: LSTM networks are composed of memory cells, which are the fundamental units that store and maintain information over long time intervals. Each memory cell has mechanisms to control the flow of information, ensuring that important data is retained and irrelevant data is discarded.

• Gates: LSTMs use three types of gates—input gate, forget gate, and output gate—to regulate the flow of information in and out of the memory cells. These gates use sigmoid activation functions to determine which information to add to the cell state, which to remove, and what to output.

  ~ Input Gate: Controls the extent to which new info flows into the memory cell.
  ~ Forget Gate: Determines which info should be discarded from the memory cell.

~ Output Gate: Regulates the output of the information stored in the memory cell.

• Cell State: The cell state acts as a conveyor belt, running through the entire LSTM network. It is manipulated by the gates, ensuring the constant flow of gradients during backpropagation and preventing the vanishing gradient problem.

II. GRU

The Gated Recurrent Unit (GRU) is a simpler variant of the LSTM, introduced by Cho et al. in 2014. The GRU’s internal structure is less complex than the LSTM, making it easier to train due to fewer computations involved. It was introduced to address some of the limitations of standard RNNs, such as the vanishing gradient problem.

The key components of the GRU have been discussed below.

• Gates: The Gated Recurrent Unit (GRU) simplifies the LSTM architecture by merging the three gating units of LSTM into two: the update gate and the reset gate.
  ~ Update Gate: This gate determines how much of the previous memory (hidden state) needs to be passed to the current state. It helps the model decide how much past information to retain.
  ~ Reset Gate: This gate decides how much of the past information to forget. It allows the model to reset its memory when necessary, helping to eliminate outdated information.

• State Update:

  ~ Unlike LSTMs, GRUs combine the cell state and hidden state into a single state. This simplifies the architecture and reduces the number of gates needed.
  ~ The GRU uses the reset gate to control how much past information to combine with the new input, and the update gate to decide the extent to which the new state should replace the previous state.

III. BRNN

The bidirectional recurrent neural network (BRNN) was proposed by (Schuster and Paliwal, 1997) to overcome the limitations of the regular (unidirectional) RNN that only utilizes past input information. BRNNs enhance the capability of RNNs by processing data in both forward and backward directions, thereby using all available input information from the past and future of a specific time frame.

• Structure: The structure of a BRNN involves splitting the state neurons into two parts: one for the positive time direction (forward states) and another for the negative time direction (backward states). These states do not interact with each other directly. This configuration allows BRNNs to consider the entire sequence of data points before and after the current point, effectively capturing context from both directions.
• Improved Context Utilization: By processing data in both directions, BRNNs can leverage future input information along with past data, providing a more comprehensive understanding of the sequence.
• Enhanced Prediction Accuracy: The ability to use information from the entire sequence leads to better prediction accuracy, particularly for tasks involving complex temporal dependencies.

Types of Recurrent Neural Networks

RNNs typically utilize a one-to-one architecture, where one input sequence is associated with one output. However, they can be adapted into various configurations to suit specific purposes. Here are several common RNN types:

I. One-To-One

A One-to-One RNN architecture involves a single input being mapped to a single output. This is the simplest form of neural network architecture and is commonly used in traditional neural networks. While RNNs are known for their ability to handle sequential data, a One-to-One architecture essentially processes data without considering temporal dependencies, making it more akin to a feedforward neural network.

The diagram below illustrates the “one-to-one” Recurrent Neural Network [Tx=Ty=1]. It can be observed that the RNN processes a single input value x and generates a single output value ŷ. The initial hidden state (a<0>) is used to process the input and is updated based on the input data.

​

Source: https://stanford.edu/

II. One-to-Many

One-to-many RNNs take a single input and produce multiple outputs. This architecture is particularly useful for applications that require generating a sequence of data from a single input. For instance, in image captioning, a single image (the input) can be used to generate a descriptive sentence (the output sequence). The process involves interpreting the image and sequentially generating the words that form the caption, ensuring that the generated text accurately describes the content of the image.

The diagram below illustrates a “One-to-Many” Recurrent Neural Network [Tx=1, Ty>1]. It can be observed, that the RNN processes the input 𝑥 to produce the first hidden state (a<0>) and the first output (ŷ<1>). The network generates a sequence of outputs over several time steps: (ŷ<1>, ŷ<2>, …, ŷ<Ty>).

Source: https://stanford.edu/

III. Many-to-Many

Many-to-many RNNs take multiple inputs and produce multiple outputs. This architecture is ideal for tasks where both the input and output are sequences of data, and the relationship between elements in the sequences must be learned and preserved. A common example is language translation, where an RNN analyzes a sentence in one language and generates a correctly structured sentence in another language

The diagram below illustrates a “Many-to-Many” Recurrent Neural Network [Tx=Ty]. It can be observed that the RNN processes each input, updating the hidden state and generating an output at each time step. The RNN receives a sequence of inputs over time steps: (x¹, x², …, x^Tx) and produces a sequence of outputs corresponding to these inputs: (ŷ¹, ŷ², …, ŷ^Ty).

​

Source: https://stanford.edu/

IV. Many-to-One

Several inputs are mapped to an output. This is helpful in applications like sentiment analysis, where the model predicts customers’ sentiments like positive, negative, and neutral from input testimonials. The diagram below illustrates a “many-to-one” Recurrent Neural Network [Tx >1, Ty=1].

It can be observed that the RNN processes a sequence of inputs (x<1>, x<2>, …, x<Tx>) over time steps. The initial hidden state (a<0>) is updated as the network processes each input in the sequence. The network produces a single output (ŷ) after processing the entire input sequence.

Source: https://stanford.edu/

Advantages of RNN

RNNs are a powerful class of neural networks that bring multiple benefits including but not limited to sequential processing, the ability to process input of any length, constant model size, weight sharing, and more.

I. Process Sequential Data

RNNs have the unique ability to retain memory of all previous inputs through their hidden states. This allows them to process sequential data effectively in time steps, a capability that other machine learning models lack. For example, when teaching a computer to read a word, each letter is processed sequentially, with the context from preceding letters influencing the interpretation of the current letter. This sequential processing makes RNNs particularly suited for tasks like language modeling, speech recognition, and time-series analysis, where understanding the order and context of the data is crucial.

II. Flexible Input Length

Recurrent Neural Networks (RNNs) offer the significant advantage of processing inputs of any length, making them highly flexible and adaptable. This capability is particularly beneficial in applications such as language translation, speech recognition, and text generation, where the input data can vary greatly in length. Unlike traditional neural networks that require fixed-size inputs, RNNs dynamically adjust to the length of the input sequence, processing data step-by-step and updating their hidden states accordingly. This allows RNNs to handle variable-length sequences efficiently without the need for extensive preprocessing.

III. Constant Model Size

One significant advantage of Recurrent Neural Networks (RNNs) is that the model size does not increase with the size of the input. This efficiency is achieved because RNNs process inputs sequentially and update their hidden states step-by-step, regardless of the input length. Unlike other models that might require expanding layers or additional parameters to handle larger inputs, RNNs maintain a consistent model architecture. This characteristic allows them to handle varying input sizes without an increase in computational complexity or memory usage, making RNNs particularly suitable for applications involving long sequences, such as text analysis, time-series forecasting, and speech recognition.

IV. Weights Sharing

Another key advantage of Recurrent Neural Networks (RNNs) is that weights are shared across time steps. This means the same set of weights processes each input in the sequence, ensuring consistency and efficiency.

• Consistency: The RNN applies the same transformations to each element of the input sequence, capturing stable patterns over time and improving generalization from training data to unseen sequences.
• Efficiency: Reduces the number of parameters the network needs to learn, simplifying the training process and making it less computationally expensive.

Weight sharing also enables RNNs to better capture temporal dynamics and relationships within data, enhancing their performance in tasks like language modeling, speech recognition, and time-series forecasting. By using the same weights repeatedly, RNNs focus on learning the underlying patterns rather than noise, reducing the risk of overfitting and achieving more accurate and reliable results.

Disadvantages of RNN

Two widely known issues with properly training recurrent neural networks are the vanishing gradient problem and the exploding gradient problem, as detailed by Bengio et al. (1994). Another challenge that follows is a “computationally slower model” compared to other neural network architectures.

I. Exploding Gradient

During initial training, an RNN may incorrectly predict outputs, requiring multiple iterations to adjust the model’s parameters and reduce the error rate. The gradient, which measures the sensitivity of the error rate to the model’s parameters, can be visualized as a slope. A steeper gradient allows for faster learning, while a shallower gradient slows down the process.

However, when the gradient increases exponentially, it leads to an exploding gradient problem, making the RNN unstable. In such cases, the gradients become excessively large, causing the RNN to behave erratically. This instability often results in performance issues such as overfitting, where the model performs well on training data but fails to generalize to real-world data.

II. Vanishing Gradient

The vanishing gradient problem occurs when a model’s gradient approaches zero during training. When this happens, the RNN fails to learn effectively from the training data, leading to underfitting. An underfit model struggles to perform well in real-life applications because its weights aren’t adjusted properly.

RNNs are particularly susceptible to vanishing and exploding gradients when processing long data sequences, making it challenging to capture dependencies over extended periods. This limitation hampers the network’s ability to model complex temporal patterns, ultimately reducing its effectiveness and accuracy in practical scenarios.

III. Computationally Slower

Recurrent Neural Networks (RNNs) are computationally slower compared to other neural network architectures primarily due to their sequential nature. In RNNs, each step in the sequence depends on the previous step, meaning the computations cannot be easily parallelized. This sequential processing significantly slows down the training and inference times, especially for long sequences.

Additionally, maintaining and updating the hidden states across time steps requires more computational resources, further adding to the time consumed during the training process. This slower processing time makes RNNs less suitable for applications requiring real-time processing where quicker training and inference times are crucial.

RNN in Trading

Stock market prediction is a crucial aspect of quantitative trading, aiming to forecast future stock prices based on historical data. Recurrent Neural Networks (RNNs) are particularly suitable for this task due to their ability to process time series data and capture dependencies over time.

I. Case Study 1

The study conducted by (Zhu, 2020) focuses on Apple’s stock (AAPL) data, spanning from August 9, 2009, to August 12, 2020. The data includes daily trends such as opening price, highest price, lowest price, and closing price. For the prediction task, only the opening prices are used. The data is normalized to the range [0, 1] to ensure consistency and suitability for training the neural network. The dataset is split into 65% for training and 35% for testing.

The model architecture used in this study consists of a two-layer RNN:

• The first layer contains 50 units.
• The second layer contains 100 units.

The network uses Mean Squared Error (MSE) as the loss function and the Adam optimizer for training. The input size is set to 1, and the timestep parameter determines the number of previous days’ prices used to predict the next day’s price. Two timestep configurations, 5 and 10, are experimented with to evaluate the model’s performance.

The RNN model is trained using Backpropagation Through Time (BPTT), adjusting network parameters by propagating error gradients backward through the sequence. The model is trained for 50 epochs, with a batch size of 64 and a dropout rate of 0.2 to prevent overfitting. The performance is evaluated using metrics such as MSE, Root Mean Squared Error (RMSE), and Mean Absolute Error (MAE).

The study concluded that the RNN model can effectively predict stock prices with high accuracy, particularly when the timestep is set to a smaller value (e.g., 5). The loss rate stabilizes after about 30 epochs, reaching approximately 0.1%. The predicted prices closely match the actual prices, demonstrating the model’s ability to capture trends and make accurate predictions.

II. Case Study 2

Another study performed by (Dey, 2021) and his colleagues evaluated the performance of different RNN models—Simple RNN, LSTM, and GRU—in predicting stock prices using datasets from Honda Motor Company (HMC), Oracle Corporation (ORCL), and Intuit Inc. (INTU). The evaluation considers one-day, three-day, and five-day intervals and uses metrics such as Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), Root Mean Squared Error (RMSE), and R-squared (R2) values.

• Simple RNN: Showed good performance but was outperformed by LSTM and GRU due to its susceptibility to the vanishing gradient problem.
• LSTM: Demonstrated improved performance, particularly for stocks with moderate fluctuations.
• GRU: Performed the best across all datasets and time intervals, effectively managing long-term dependencies due to its simpler architecture.

GRU achieved the highest R2 values, indicating a better fit to the regression model. For moderately fluctuating data, LSTM performed slightly better, while for slightly fluctuating data, GRU again outperformed the others.

The Bottom Line

Recurrent Neural Networks (RNNs) are powerful neural network architectures designed for sequential data processing by leveraging their ability to maintain memory of previous inputs. This unique characteristic makes them particularly effective for tasks where the order of data points is crucial, such as speech recognition, language modeling, and time-series forecasting. RNNs achieve this through the use of hidden states that are influenced by both current inputs and past information, allowing them to capture temporal dependencies. Advanced variants like LSTMs and GRUs address the limitations of traditional RNNs, such as the vanishing gradient problem, by incorporating mechanisms to retain information over long periods, enhancing their capability to model complex sequences.