LSTM Explained: From Basic Concepts to a Complete Practical Project

LSTM (Long Short-Term Memory)

What is it?

LSTM is a special type of Recurrent Neural Network (RNN) designed to learn from sequences of data (text, audio, time series) while remembering important information over long periods. It was introduced in 1997.

Think of it like a person reading a story: you don’t forget the main character’s name by page 50, and you also don’t try to remember every tiny detail. You keep what matters and discard what doesn’t. LSTM does something similar.

Why did it come? (The problem it solved)

Regular RNNs process data step by step and pass a “hidden state” forward. In theory they can remember the past, but in practice they have a big flaw called the vanishing gradient problem.

Vanishing Gradient Problem

1. A neural network learns by sending error signals (gradients) backward through its layers or time steps to update its weights.
2. In deep networks and long sequences, these signals are multiplied by small numbers again and again.
3. The signal shrinks (the gradient value gets smaller and smaller (like 0.5 → 0.25 → 0.12 → almost 0) until it is too tiny to help the network learn) toward zero by the time it reaches the early steps.
4. As a result, the early parts of the network barely learn anything.
5. This is why plain RNNs forget long-term information, and LSTM was designed to fix it.

• During training, error signals are sent backward through many time steps.
• These signals get multiplied again and again by small numbers, so they shrink toward zero.
• Result: the network can’t learn connections between things that are far apart (e.g., a word at the start of a long sentence and one at the end).

LSTM was created to fix this by giving the network a dedicated memory path that information can flow along with little interference.

How does it work?

An LSTM cell has two kinds of memory and three “gates”:

• Cell state: the long-term memory, like a conveyor belt running through time.
• Hidden state: the short-term output at each step.

The gates (each uses a sigmoid function, outputting values between 0 and 1) control the flow:

1. Forget gate: decides what to erase from the cell state.
2. Input gate: decides what new information to store.
3. Output gate: decides what part of memory to output right now.

In simple form:

• Forget: f = σ(W_f·[h_prev, x] + b_f)
• Input: i = σ(W_i·[h_prev, x] + b_i), candidate c̃ = tanh(W_c·[h_prev, x] + b_c)
• Update memory: c = f·c_prev + i·c̃
• Output: o = σ(W_o·[h_prev, x] + b_o), h = o·tanh(c)

Because the memory update is mostly additive (not repeated multiplication), gradients can flow back much further without vanishing.

Where and how is it used?

Natural language

• Machine translation
• Text generation
• Sentiment analysis
• Chatbots

Speech

• Speech recognition (e.g., early Google Voice, Siri)
• Speech synthesis

Time series

• Stock/price forecasting
• Weather prediction
• Energy demand
• Sensor data

Healthcare

• ECG/EEG analysis
• Patient outcome prediction

Video and vision

• Action recognition
• Image captioning (combined with CNNs)

Music

• Generating melodies
• Music transcription

Anomaly detection

• Fraud detection
• Machine failure prediction
• Network intrusion

Handwriting

• Handwriting recognition and generation

Kinds of problems it can solve

• Sequence → one output: classify a review as positive/negative.
• Sequence → sequence: translate a sentence from English to Urdu.
• One input → sequence: generate a caption from an image.
• Forecasting: predict the next value(s) in a series.

Best suited when order matters and context from earlier steps affects later ones.

Strengths and limitations

Strengths

• Handles long-range dependencies much better than plain RNNs
• Works with variable-length sequences
• Proven and widely supported (TensorFlow, PyTorch, Keras)

Limitations

• Slow to train because it processes sequentially (hard to parallelize)
• More parameters than a simple RNN
• Still struggles with extremely long sequences
• Largely replaced by Transformers (GPT, BERT, etc.) for most NLP tasks

Related variants

• GRU: a simpler, faster cousin with 2 gates.
• Bidirectional LSTM: reads the sequence forward and backward.
• Stacked LSTM: multiple LSTM layers for deeper learning.

Hyperparameters that matter

• Hidden units (e.g., 32, 64, 128): more units can learn more complex patterns but raise the risk of overfitting and slow training.
• Number of layers: stacking 2 to 3 layers is common; more than that rarely helps and is hard to train.
• Sequence length (time steps): how far back the model looks. Too short loses context, too long adds cost and noise.
• Batch size and learning rate: Adam with a learning rate around 0.001 is a common starting point.
• Dropout: randomly switches off units during training to reduce overfitting (typically 0.2 to 0.5).

Data preparation

• Scale your data. LSTMs work poorly with raw large numbers. Use normalization (0 to 1) or standardization.
• Input shape is 3D: (samples, time steps, features). Getting this wrong is the most common beginner error.
• Padding and masking: when sequences have different lengths, pad them to equal length and use masking so padding is ignored.
• Split by time, not randomly, for time-series data. Random splits leak future information into training.

Training problems and fixes

Overfitting

• Use dropout
• Use early stopping
• Add more data
• Use fewer units

Exploding gradients

• Use gradient clipping (e.g., clip norm = 1.0)

Slow training

• Use smaller sequences
• Use a GPU
• Use the cuDNN-optimized layer

Poor results

• Check data scaling
• Tune the sequence length
• Try GRU

Practical tips

• Start simple. One LSTM layer with 32 to 64 units is often enough. Add complexity only if needed.
• Compare against a baseline. For time series, always test against a naive forecast (such as “tomorrow equals today”). Many LSTM models fail to beat it.
• Use early stopping to halt (halt) training when validation loss stops improving.
• Return sequences: return_sequences=True outputs a value at every step (needed when stacking LSTMs or for sequence-to-sequence tasks), while False outputs only the final step.

Common mistakes

• Stateful vs. stateless: by default, LSTM resets memory between batches. Stateful mode carries memory across batches, useful for very long continuous streams.

Core concept

Stateless: each input is processed independently, and no information carries over between calls.

Stateful: a persistent state (memory) is carried across calls or segments, so earlier inputs influence later outputs.

1. Feeding unscaled data
2. Wrong input shape
3. Randomly shuffling time-series data before splitting
4. Using too many units on a small dataset
5. Judging only by training loss instead of validation loss
6. Expecting LSTM to predict inherently random things (like stock prices) accurately

When to choose LSTM vs. alternatives

• LSTM: small to medium sequence data, time series, limited compute, real-time or on-device use.
• GRU: similar accuracy with faster training; worth trying first on smaller datasets.
• 1D CNN or Temporal CNN: faster, good for local patterns in sensor or signal data.
• Transformer: large datasets, long text, or when you can use pretrained models.
• Classical methods (ARIMA, XGBoost): often competitive or better for simple tabular or seasonal time-series.

Evaluation metrics

• Regression/forecasting: MAE, RMSE, MAPE
• Classification: accuracy, precision, recall, F1-score
• Text generation or translation: perplexity, BLEU

Perplexity

Measures how well an LSTM language model predicts the next word.
Lower is better. A low value means the model is less “surprised” by the real text.
Used for LSTM text generation and language modeling.

BLEU

Measures how close a translated (or generated) sentence is to a human reference sentence.
Higher is better (0 to 1, or 0 to 100).
Used for LSTM translation models (sequence-to-sequence with LSTM encoder and decoder).

Frameworks

• Keras/TensorFlow: easiest for beginners
• PyTorch: flexible and popular in research
• scikit-learn: does not provide LSTM, so you need one of the above

Key limitations to remember

• It is a “black box,” so explaining why it made a prediction is hard.
• Needs a reasonable amount of data; it can underperform on very small datasets.
• Results vary between runs because of random weight initialization, so set a random seed for reproducibility.

The big picture

LSTM, Transformers, and LLMs are all part of the same family: models that handle sequences (mostly text). They are steps in an evolution, each trying to solve the weaknesses of the one before.

Simple RNN → LSTM → LSTM + Attention → Transformer → LLM

LSTM’s weakness that led to Transformers

LSTM improved memory, but two problems remained:

1. Sequential processing: it reads one word at a time, so it can’t use the full power of GPUs to process everything in parallel. Training on huge datasets was slow.
2. Information bottleneck: in translation, the LSTM had to squeeze an entire sentence into one fixed-size vector. Long sentences lost details.Bottleneck means a very narrow place where too much information has to pass through, so things slow down or get lost.
3. Example: Imagine pouring a big bucket of water into a bottle with a very thin neck. Water cannot go through quickly, and some of it spills.

LSTM:

The model reads a long sentence.

It must save the full meaning of the sentence in one small box (a fixed-size vector).

If the sentence is very long, the small box cannot hold everything.

So the model forgets some words and details.

“A bottleneck is a small space that cannot hold all the information, so some information is lost.”

The bridge: Attention

Attention was first added on top of LSTMs to fix the bottleneck. Instead of relying on one summary vector, the model could “look back” at every word of the input and focus on the most relevant ones.

Example: when translating a sentence, for each output word the model asks, “Which input words matter most right now?”

This worked so well that researchers asked: what if we remove the LSTM entirely and use only attention? That idea became the Transformer.

What a Transformer changed

Processing

• LSTM: one step at a time
• Transformer: all words at once (parallel)

Memory method

• LSTM: cell state + gates
• Transformer: self-attention

Long-range links

• LSTM: good, but fades with distance
• Transformer: any word can directly connect to any other word, the farther a word is from the current word, the less the model remembers it.

Training speed

• LSTM: slow on large data
• Transformer: fast on GPUs

Scaling to huge data

• LSTM: difficult
• Transformer: very easy

The core idea is self-attention: every word looks at every other word in the sentence and decides how much to focus on each. This replaces the job that LSTM’s gates and cell state did.

How LLMs fit in

An LLM (Large Language Model) such as GPT, Claude, or Gemini is simply a very large Transformer trained on huge amounts of text to predict the next word (token).

• Transformers made it practical to scale up to billions of parameters and massive datasets.
• Before this, LSTM-based language models existed (such as ELMo and early Google Translate), but they were smaller and could not scale the same way.

So: LSTM was the leading language model architecture, Transformers replaced it, and LLMs are what you get when you scale Transformers massively.

What is the same in both

Even though they work differently, they share core ideas:

• Both process sequences and care about context
• Both use learned weights trained with backpropagation
• Both can predict the next item in a sequence
• Both use embeddings (turning words into vectors)
• The LSTM gates idea (controlling what information flows) lives on in Transformers through attention weights and gating inside feed-forward layers

Is LSTM still useful?

Yes, in some cases:

• Small datasets and limited compute
• Real-time or on-device applications (low memory)
• Simple time-series forecasting
• Streaming data where you process one step at a time

Newer research also revisits LSTM-style ideas. Models like xLSTM and state-space models (such as Mamba) try to combine the efficiency of recurrent models with the quality of Transformers.

Summary

LSTM taught us how to give a network memory. Attention taught us how to look at the right parts of a sequence. Transformers dropped recurrence and kept only attention, which made training parallel and scalable. LLMs are Transformers scaled up enormously. Understanding LSTM makes it much easier to understand why Transformers were invented.

Complete LSTM Project: From Importing Data to Testing on New Data

This is a time-series forecasting example (predicting the next value from the previous 30). Each step is a separate code block, so you can copy and run them in order in Jupyter, Colab, or a .py file.

Step 0: Install libraries (once)

bash

pip install tensorflow pandas numpy scikit-learn matplotlib joblib

Step 1: Import libraries

python

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import joblib

from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import mean_absolute_error, mean_squared_error
import tensorflow as tf
from tensorflow.keras.models import Sequential, load_model
from tensorflow.keras.layers import Input, LSTM, Dense, Dropout
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint
# Reproducibility
np.random.seed(42)
tf.random.set_seed(42)

np.random.seed(42): It fixes the random numbers used by NumPy, so you get the same results every time you run the code.

tf.random.set_seed(42): It fixes the random numbers used by TensorFlow (like the starting weights of the model), so training gives similar results every time.

Note: The number 42 is just a choice. You can use any number, but keep it the same to repeat your results.

Step 2: Load the dataset

Your CSV needs a date column and a value column. Change the names below to match your file.

DATE_COL = "Date"
VALUE_COL = "Temp"
WINDOW = 30

DATE_COL = "Date"
It stores the name of the date column in the file.

VALUE_COL = "Temp"
It stores the name of the column we want to predict (the temperature).

WINDOW = 30
It tells the model to look at the past 30 days to predict the next day.

Why keep them at the top?
If you use another dataset, you only change these 3 lines, and the rest of the code stays the same.

Simple line for your writing:
“These three variables store the column names and the number of past days the model uses, so the code can be easily changed for any dataset.”

url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/daily-min-temperatures.csv"

df = pd.read_csv(url, parse_dates=[DATE_COL])
df[VALUE_COL] = pd.to_numeric(df[VALUE_COL], errors="coerce")  # safety: bad values become NaN
df = df.sort_values(DATE_COL).reset_index(drop=True)

parse_dates: It tells pandas to read the Date column as real dates (not plain text), so we can sort and plot by time.

errors="coerce": If any value is bad (like text instead of a number), it is changed to an empty value (NaN) instead of stopping the code with an error.

Keep the last 365 days aside as the “new dataset” for the final test (Step 13)

new_df = df.tail(365)
df = df.iloc[:-365].reset_index(drop=True)

df.to_csv("data.csv", index=False)
new_df.to_csv("new_data.csv", index=False)

print("Training data:", df.shape)
print("New (unseen) data:", new_df.shape)

How to view the dataset

Run these after loading. In a .py file, keep the print(). In Jupyter or Google Colab, you can write just df.head() on its own line, and it shows a nicely formatted table.

# First 5 rows
print(df.head())
# Last 5 rows
print(df.tail())
# Column names, data types, and row count
print(df.info())
# Basic statistics (min, max, mean, etc.)
print(df.describe())
# Missing values
print("Missing values:", df[VALUE_COL].isna().sum())

Graphical representation

# Plot it
plt.figure(figsize=(10, 4))
plt.plot(df[DATE_COL], df[VALUE_COL])
plt.title("Daily minimum temperature")
plt.xlabel("Date")
plt.ylabel("Temperature (°C)")
plt.show()

Step 3: Clean and explore

python

interpolate() fills missing values (NaN) by guessing them from the values around them.

Example: if the temperatures are 10, NaN, 14, then interpolate fills the gap with 12 (the middle value).

Why use it here? In time series, we cannot just delete a row, because it would break the time order. Filling the gap keeps the data continuous.

# Fill small gaps; drop anything still missing
df[VALUE_COL] = df[VALUE_COL].interpolate()
df = df.dropna(subset=[VALUE_COL]).reset_index(drop=True)

“reset_index(drop=True) gives the rows fresh numbers from 0 after sorting or deleting, so the data stays clean and in order.”

plt.figure(figsize=(10, 4))
plt.plot(df[DATE_COL], df[VALUE_COL])
plt.title("Original data")
plt.show()
values = df[[VALUE_COL]].values   # shape (n, 1)

Step 4: Split by time (not randomly)

python

n = len(values)
train_end = int(n * 0.70)
val_end = int(n * 0.85)

train = values[:train_end]
val = values[train_end:val_end]
test = values[val_end:]
print("Train:", train.shape, "Val:", val.shape, "Test:", test.shape)

Total: 3285
Train: (2299, 1) Val: (493, 1) Test: (493, 1)

Step 5: Scale the data

Fit the scaler on the training data only. Fitting on everything leaks information from the future.

Get Mohammad Atif’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

python

scaler = MinMaxScaler()
scaler.fit(train)

train_s = scaler.transform(train)
val_s = scaler.transform(val)
test_s = scaler.transform(test)

fit() learns the minimum and maximum values from the training data.

transform() uses those learned values to convert the data into the 0 to 1 range.

Why two steps on train?

First, the scaler must learn the min and max (fit). Then it can apply them (transform).

We fit only on train because val and test are “future” data. If the scaler learned from them, information from the future would leak into training (called data leakage).

Val and test only use transform, with the min and max learned from train.

Step 6: Create sequences

This turns the data into the 3D shape LSTM needs: (samples, time steps, features).

def make_sequences(data, window):
    X, y = [], []
    for i in range(window, len(data)):
        X.append(data[i - window:i])
        y.append(data[i])
    return np.array(X), np.array(y)

Purpose: It converts one long list of values into many small examples (past values → next value) that the LSTM can learn from.

How it works (with WINDOW = 3):

Data: 10, 20, 30, 40, 50

Example 1: X = [10, 20, 30] → y = 40

Example 2: X = [20, 30, 40] → y = 50

In the code:

X = the past values (the input)

y = the next value (the answer the model must predict)

data[i - window:i] takes the past 30 values

data[i] takes the value right after them

Why it is needed: LSTM cannot learn from a plain list. It needs the 3D shape (samples, time steps, features), and this function creates it.

# Add the last WINDOW rows of the previous split so the first
# validation/test samples have enough history
X_train, y_train = make_sequences(train_s, WINDOW)

X_val, y_val = make_sequences(np.concatenate([train_s[-WINDOW:], val_s]), WINDOW)
X_test, y_test = make_sequences(np.concatenate([val_s[-WINDOW:], test_s]), WINDOW)
print("X_train:", X_train.shape)   # (samples, 30, 1)
print("X_val:  ", X_val.shape)
print("X_test: ", X_test.shape)

Example (WINDOW = 3):

Train ends with: 8, 9, 10

Val is: 11, 12, 13

Joined data: 8, 9, 10, 11, 12, 13

First example: X = [8, 9, 10] → y = 11

Step 7: Build the model

Most Important Layers in LSTM

1. LSTM layer (the most important)

It learns patterns from the sequence and keeps memory using its gates and cell state.

2. Dense layer

It gives the final output, such as the predicted value or class.

3. Dropout layer

It switches off some units randomly during training to stop overfitting.

4. Input layer

It defines the shape of the data: (time steps, features).

5. Embedding layer (for text tasks only)

It turns words into numbers so the LSTM can read them.

Optional layers

Bidirectional: reads the sequence forward and backward.

Stacked LSTM: uses more than one LSTM layer for deeper learning.

model = Sequential([
    Input(shape=(WINDOW, 1)),
    LSTM(64),
    Dropout(0.2),
    Dense(1)
])

It builds the LSTM model by stacking layers one after another, like a pipeline. Data enters at the top and the prediction comes out at the bottom.

Sequential([...])

• It means the layers are stacked in order, one after the other.

Input(shape=(WINDOW, 1))

• It tells the model what data to expect: 30 past days, with 1 value per day.

LSTM(64)

• It reads the 30 days, learns the pattern, and keeps memory using 64 units.

Dropout(0.2)

• It switches off 20% of units randomly during training, so the model does not memorize the data.

Dense(1)

• It gives the final answer: one predicted value (the next day’s temperature).

Flow in one line:
30 days of data → LSTM learns the pattern → Dropout prevents overfitting → Dense gives 1 prediction.

model.compile(optimizer="adam", loss="mse")

It sets up how the model will learn before training starts. The model is built, but this line tells it how to improve and how to measure its mistakes.

optimizer="adam"

• The optimizer is the method that updates the model’s weights to reduce mistakes.
• Adam is a popular choice because it adjusts the learning speed automatically and works well in most cases.

loss="mse"

• Loss is the score that shows how wrong the predictions are.
• MSE (Mean Squared Error) takes the difference between the predicted and real value, squares it, and averages it.
• A lower value means better predictions.

Example:

• Real temperature = 10, predicted = 12
• Error = 2, squared error = 4
• The model tries to make this number smaller after every step.

model.summary()

Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ lstm (LSTM)                     │ (None, 64)             │        16,896 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 64)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense (Dense)                   │ (None, 1)              │            65 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
 Total params: 16,961 (66.25 KB)
 Trainable params: 16,961 (66.25 KB)
 Non-trainable params: 0 (0.00 B)

Param #

• Parameters are the numbers (weights) the model learns during training.
• LSTM = 16,896: it has the most because of the four internal parts (three gates and the candidate memory).
• Dropout = 0: it has nothing to learn, it only switches off units.
• Dense = 65: 64 weights (one for each LSTM unit) + 1 bias.

Total params: 16,961

• All learnable numbers added together: 16,896 + 0 + 65. The size is about 66 KB, so this is a very small model.

Trainable params: 16,961

• These numbers are updated during training. Here, all of them are trainable.

Non-trainable params: 0

• Fixed numbers that do not change during training. Here there are none.

“The model summary shows each layer, its output shape, and the number of parameters, and our LSTM model has 16,961 trainable parameters.”

Step 8: Train with early stopping

callbacks = [
    EarlyStopping(monitor="val_loss", patience=10, restore_best_weights=True),
    ModelCheckpoint("best_model.keras", monitor="val_loss", save_best_only=True)
]

Callbacks: The Idea

A callback is a helper that watches the model during training and takes action automatically, such as stopping early or saving the best version. You don’t have to watch the training yourself.

EarlyStopping

• Purpose: stops training when the model stops improving, so it does not start memorizing the data (overfitting).
• monitor="val_loss": it watches the validation loss, which shows how well the model does on data it has not trained on.
• patience=10: it waits 10 more rounds (epochs) for improvement. If there is none, it stops.
• restore_best_weights=True: when it stops, it goes back to the best version of the model, not the last one.

ModelCheckpoint

• Purpose: saves the model to a file during training, so the best version is never lost.
• "best_model.keras": the file name where it is saved.
• save_best_only=True: it saves only when the validation loss gets better than before.

Simple example:

• Validation loss gets better until epoch 20, then stops improving.
• EarlyStopping waits until epoch 30, then stops training.
• The model goes back to the weights from epoch 20.
• ModelCheckpoint has already saved epoch 20 in best_model.keras.

Why use both?

• EarlyStopping decides when to stop and which weights to use in memory.
• ModelCheckpoint keeps a saved copy on disk.

“Callbacks watch the validation loss during training. EarlyStopping stops training when the model stops improving, and ModelCheckpoint saves the best version of the model to a file.”

python

history = model.fit(
    X_train, y_train,
    validation_data=(X_val, y_val),
    epochs=100,
    batch_size=32,
    shuffle=False,
    callbacks=callbacks,
    verbose=1
)

“shuffle=False keeps the data in time order during training, because in a time series the order of the values matters.”

Step 9: Check the training curves

python

plt.figure(figsize=(8, 4))
plt.plot(history.history["loss"], label="Train loss")
plt.plot(history.history["val_loss"], label="Validation loss")
plt.legend()
plt.title("Loss during training")
plt.show()

If training loss keeps falling while validation loss rises, the model is overfitting. Use fewer units, more dropout, or more data.

This is the loss curve. It shows how the model’s mistakes (loss) changed during 100 rounds of training (epochs).

Blue line: Train loss

• Mistakes on the data the model is learning from.
• It starts high (about 0.037) and drops very fast in the first few epochs.

Orange line: Validation loss

• Mistakes on data the model has not trained on.
• It starts low (about 0.011) and goes down slowly.

What it tells us

• The model is learning: both lines go down.
• No overfitting: the validation loss does not go up. It stays flat and low.
• Training is complete: after about epoch 30 to 40, both lines become almost flat, so the model has stopped improving much.
• Both losses are close (about 0.009 and 0.007), so the model works well on new data too.

Why is validation loss lower than train loss?
This looks strange, but it is normal here. Dropout is switched on while training (which makes train loss higher) and switched off during validation. Also, the validation data may simply be easier to predict.

One note: the lines run until epoch 100, so early stopping did not trigger. The model kept improving a tiny bit until the end.

“The loss curve shows that both training and validation loss decrease and become stable, with no sign of overfitting, so the model has learned the pattern well.”

Step 10: Test on the test dataset

pred_s = model.predict(X_test)

# Convert back to the original scale
pred = scaler.inverse_transform(pred_s)
actual = scaler.inverse_transform(y_test)
mae = mean_absolute_error(actual, pred)
rmse = np.sqrt(mean_squared_error(actual, pred))
print(f"LSTM  -> MAE: {mae:.4f} | RMSE: {rmse:.4f}")
# Baseline: "tomorrow = today"
naive_mae = mean_absolute_error(actual[1:], actual[:-1])
naive_rmse = np.sqrt(mean_squared_error(actual[1:], actual[:-1]))
print(f"Naive -> MAE: {naive_mae:.4f} | RMSE: {naive_rmse:.4f}")

plt.figure(figsize=(10, 4))
plt.plot(actual, label="Actual")
plt.plot(pred, label="Predicted")
plt.legend()
plt.title("Test set: actual vs predicted")
plt.show()

If the LSTM does not beat the naive baseline, the model is not adding value yet. Try tuning the window size, units, or features.

This is the test result. It compares the real temperatures (blue) with the temperatures the LSTM predicted (orange) on data the model never saw during training.

Blue line: Actual

• The real temperature values. It goes up and down a lot from day to day.

Orange line: Predicted

• The values predicted by the LSTM.

What it tells us

• The model learned the main pattern: the orange line follows the same yearly wave as the blue line (warm period, then cold around 300, then warm again).
• Predictions are smoother: the orange line does not reach the extreme highs and lows. For example, around 110 the real value is about 24, but the model predicts about 19.
• Daily jumps are hard to predict: sudden changes in one day are random, so the model gives a safe, average-like answer.

Why this is normal

• LSTM learns the trend and pattern, but it cannot predict random sudden spikes.
• The model is good for the overall direction, not for exact daily extremes.

Important check: compare the MAE and RMSE of the LSTM with the naive baseline (Step 10). That number tells you if the LSTM is really better than just saying “tomorrow = today.”

“The graph shows that the predicted values follow the overall pattern of the actual values, but the model gives smoother results and misses sudden peaks and drops.”

Step 11: Save the model and scaler

You need both. The scaler is required to prepare new data the same way.

python

model.save("lstm_model.keras")
joblib.dump(scaler, "scaler.pkl")

print("Saved: lstm_model.keras and scaler.pkl")

Step 12: Load them later (new session or new machine)

python

model = load_model("lstm_model.keras")
scaler = joblib.load("scaler.pkl")
WINDOW = 30   # must match what you trained with

Step 13: Test on a new dataset

The new file must have the same column, the same units, and at least WINDOW + 1 rows.

new_df = pd.read_csv("new_data.csv", parse_dates=[DATE_COL])
new_df = new_df.sort_values(DATE_COL).reset_index(drop=True)
new_df[VALUE_COL] = new_df[VALUE_COL].interpolate()
new_values = new_df[[VALUE_COL]].values
new_values

# Use transform only, never fit again
new_s = scaler.transform(new_values)
X_new, y_new = make_sequences(new_s, WINDOW)
new_pred = scaler.inverse_transform(model.predict(X_new))
new_actual = scaler.inverse_transform(y_new)
print("New data MAE :", mean_absolute_error(new_actual, new_pred))
print("New data RMSE:", np.sqrt(mean_squared_error(new_actual, new_pred)))

11/11 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step
New data MAE : 1.7381527425282035
New data RMSE: 2.255542747954371

plt.figure(figsize=(10, 4))
plt.plot(new_actual, label="Actual")
plt.plot(new_pred, label="Predicted")
plt.legend()
plt.title("New dataset: actual vs predicted")
plt.show()

Step 14: Predict the next value (no actual answer available)

python

last_window = scaler.transform(new_values[-WINDOW:])
last_window = last_window.reshape(1, WINDOW, 1)

next_value = scaler.inverse_transform(model.predict(last_window))
print("Predicted next value:", next_value[0][0])

Suggested folder layout

• data.csv: training data
• new_data.csv: unseen data for final testing
• train.py or notebook: Steps 1 to 11
• predict.py: Steps 12 to 14
• lstm_model.keras and scaler.pkl: saved outputs

Common errors and fixes

• Shape error: check that the input is (samples, WINDOW, 1).
• Poor results on new data: its values may be outside the training range (for example, much higher prices). The scaler and model only know the range they were trained on.
• FileNotFoundError: check the file path and your working folder.
• Different results each run: set the random seeds (done in Step 1) and expect small variation anyway.

I hope this practical guide helped you build your first LSTM. If you found it useful, please share it with friends and classmates who are starting their AI journey.