Recurrent Neural Networks (RNNs)

While Convolutional Neural Networks (CNNs) are designed to process images,
Recurrent Neural Networks (RNNs) are designed for sequence-based data —
data that changes with time or depends on previous steps. This includes text, speech,
music, time-series signals, financial data, and even human language.

RNNs are incredibly important in deep learning, especially in the field of
Natural Language Processing (NLP) — where machines understand, interpret,
and generate human language.

📌 Why Do We Need RNNs?

Traditional neural networks (MLPs and CNNs) treat each input independently.
But many real-world tasks depend on context or history.

Examples:

• Speech: Each word depends on previous words
• Text: Sentence meaning depends on earlier words
• Stock market: Today’s price depends on previous days
• Weather: Tomorrow’s prediction depends on previous patterns
• Music: Notes are played in sequence

CNNs cannot handle this type of sequential relationship.
RNNs were created to solve this.

⭐ What Is an RNN?

An RNN is a neural network with a “memory.”
It processes one step at a time and stores information from previous steps.

RNN formula (simplified):

hidden_state_t = activation( Wxh * x_t + Whh * hidden_state_(t-1) )
output_t = Why * hidden_state_t
    

This means every prediction depends on:

• Current input
• Previous hidden state (memory)

📌 Real-Life Example: Predicting the Next Word

If you type “I love deep…”, your keyboard predicts:

• learning
• neural networks
• science

RNNs understand the context of previous words to predict the next one.

⭐ Understanding Sequences & Timesteps

RNNs process data in timesteps.

For example, the sentence:

“Deep learning is amazing.”

is processed as:

• t1 → Deep
• t2 → learning
• t3 → is
• t4 → amazing

At each step, the RNN updates its memory.

📌 Limitations of Basic RNNs

RNNs are powerful, but they have one big problem:

The Vanishing Gradient Problem

When sequences are long, the gradients shrink and the network “forgets” older information.

For example, in a long paragraph, RNNs might forget the subject of the sentence.

⭐ Types of RNNs

• Simple RNN
• LSTM (Long Short-Term Memory)
• GRU (Gated Recurrent Unit)
• Bidirectional RNN
• Deep RNNs

Among these, LSTM and GRU are most popular — because they solve
the vanishing gradient problem.

📌 Real-Life Use Cases of RNNs

• Text generation (ChatGPT predecessor models)
• Language translation (Google Translate)
• Speech-to-text systems
• Voice assistants (Siri, Alexa)
• Stock price prediction
• Weather forecasting
• Music generation
• Sentiment analysis (positive/negative reviews)

⭐ RNN Architecture Example (Keras)

from tensorflow import keras
from keras import layers

model = keras.Sequential([
    layers.SimpleRNN(64, return_sequences=True),
    layers.SimpleRNN(32),
    layers.Dense(1, activation='sigmoid')
])
    

This model can classify sequential data like text or time-series.

📌 Real-Life Example: Sentiment Analysis

An RNN can read a movie review like:

“The movie was fantastic and emotional!”

and determine whether it’s:

• Positive
• Negative
• Neutral

📌 Bidirectional RNNs

These read input in both directions:

• Forward → left to right
• Backward → right to left

Useful for understanding full sentence meaning.

📌 RNNs vs CNNs

• CNNs → best for images
• RNNs → best for sequences

Example:

• CNN → detect a cat in an image
• RNN → generate a caption: “A cat sitting on a sofa.”

📌 Summary

RNNs brought a revolution in sequence-based deep learning.
They can process text, speech, and time-series by using memory of past inputs.
In the next chapter, you will learn LSTM (Long Short-Term Memory) —
a powerful version of RNNs designed to solve memory issues.