Recurrent Neural Networks (RNNs)
While Convolutional Neural Networks (CNNs) are designed to process images,
Recurrent Neural Networks (RNNs) are designed for sequence-based data —
data that changes with time or depends on previous steps. This includes text, speech,
music, time-series signals, financial data, and even human language.
RNNs are incredibly important in deep learning, especially in the field of
Natural Language Processing (NLP) — where machines understand, interpret,
and generate human language.
📌 Why Do We Need RNNs?
Traditional neural networks (MLPs and CNNs) treat each input independently.
But many real-world tasks depend on context or history.
Examples:
- Speech: Each word depends on previous words
- Text: Sentence meaning depends on earlier words
- Stock market: Today’s price depends on previous days
- Weather: Tomorrow’s prediction depends on previous patterns
- Music: Notes are played in sequence
CNNs cannot handle this type of sequential relationship.
RNNs were created to solve this.
⭐ What Is an RNN?
An RNN is a neural network with a “memory.”
It processes one step at a time and stores information from previous steps.
RNN formula (simplified):
hidden_state_t = activation( Wxh * x_t + Whh * hidden_state_(t-1) )
output_t = Why * hidden_state_t
This means every prediction depends on:
- Current input
- Previous hidden state (memory)
📌 Real-Life Example: Predicting the Next Word
If you type “I love deep…”, your keyboard predicts:
- learning
- neural networks
- science
RNNs understand the context of previous words to predict the next one.
⭐ Understanding Sequences & Timesteps
RNNs process data in timesteps.
For example, the sentence:
“Deep learning is amazing.”
is processed as:
- t1 → Deep
- t2 → learning
- t3 → is
- t4 → amazing
At each step, the RNN updates its memory.
📌 Limitations of Basic RNNs
RNNs are powerful, but they have one big problem:
The Vanishing Gradient Problem
When sequences are long, the gradients shrink and the network “forgets” older information.
For example, in a long paragraph, RNNs might forget the subject of the sentence.
⭐ Types of RNNs
- Simple RNN
- LSTM (Long Short-Term Memory)
- GRU (Gated Recurrent Unit)
- Bidirectional RNN
- Deep RNNs
Among these, LSTM and GRU are most popular — because they solve
the vanishing gradient problem.
📌 Real-Life Use Cases of RNNs
- Text generation (ChatGPT predecessor models)
- Language translation (Google Translate)
- Speech-to-text systems
- Voice assistants (Siri, Alexa)
- Stock price prediction
- Weather forecasting
- Music generation
- Sentiment analysis (positive/negative reviews)
⭐ RNN Architecture Example (Keras)
from tensorflow import keras
from keras import layers
model = keras.Sequential([
layers.SimpleRNN(64, return_sequences=True),
layers.SimpleRNN(32),
layers.Dense(1, activation='sigmoid')
])
This model can classify sequential data like text or time-series.
📌 Real-Life Example: Sentiment Analysis
An RNN can read a movie review like:
“The movie was fantastic and emotional!”
and determine whether it’s:
- Positive
- Negative
- Neutral
📌 Bidirectional RNNs
These read input in both directions:
- Forward → left to right
- Backward → right to left
Useful for understanding full sentence meaning.
📌 RNNs vs CNNs
- CNNs → best for images
- RNNs → best for sequences
Example:
- CNN → detect a cat in an image
- RNN → generate a caption: “A cat sitting on a sofa.”
📌 Summary
RNNs brought a revolution in sequence-based deep learning.
They can process text, speech, and time-series by using memory of past inputs.
In the next chapter, you will learn LSTM (Long Short-Term Memory) —
a powerful version of RNNs designed to solve memory issues.
