Building a Spam Message Classifier Using Machine Learning:

Introduction

In this article, we will build a machine learning classification model to identify spam and normal text messages.

We will use the SMS Spam Collection dataset from the UCI Machine Learning Repository. We will first load the data, explore it, and prepare it for machine learning.

Step 1: Install and Import Libraries

First, we need to install Scikit-learn and import the required libraries.

!pip install scikit-learn -q

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split

Why do we need these libraries?

• Pandas helps us read and analyze the dataset.
• NumPy supports numerical operations.
• Matplotlib creates charts and graphs.
• Seaborn helps us create statistical visualizations.
• train_test_split will divide the dataset into training and testing sets later.

Step 2: Download and Read the Dataset

We will download the SMS Spam Collection dataset from the UCI Machine Learning Repository.

import urllib.request
import zipfile

url = "https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip"
urllib.request.urlretrieve(url, "sms_spam.zip")
with zipfile.ZipFile("sms_spam.zip", "r") as zip_ref:
    zip_ref.extractall("sms_spam_data")

The dataset is downloaded as a ZIP file. We extract its contents so we can access the SMS messages.

Now, let’s read the dataset using Pandas.

df = pd.read_csv(
    "sms_spam_data/SMSSpamCollection",
    sep="\t",
    header=None,
    names=["label", "message"],
    encoding="utf-8"
)

df.head()

What does this code do?

• sep="\t" tells Pandas that the values are separated by tabs.
• header=None means the original file does not contain column names.
• names=["label", "message"] assigns names to our two columns.
• encoding="utf-8" helps read text characters correctly.
• df.head() displays the first five rows.

Our dataset contains two columns:

• label: identifies a message as spam or ham.
• message: contains the actual text message.

Here, spam means an unwanted or suspicious message, while ham means a normal message.

Step 3: Explore the Dataset

Before training a machine learning model, we need to understand our data.

Check the Number of Rows and Columns

print("Number of rows:", df.shape[0])
print("Number of columns:", df.shape[1])

df.shape[0] returns the number of rows, while df.shape[1] returns the number of columns.

This helps us understand the size of our dataset.

Check the Data Types and Information

df.info()

This command shows the column names, data types, number of non-null values, and memory usage.

It helps us identify possible data quality issues.

Display Random Messages

df.sample(5, random_state=42)

This command randomly selects five rows from the dataset.

The random_state=42 setting makes the selection reproducible, so we get the same sample when we run the code again on the unchanged dataset.

Step 4: Check Missing Values

Missing values can affect data analysis and model training.

Let’s check whether our dataset contains any missing values.

print("Missing values:\n", df.isnull().sum(), sep="")

This command counts the missing values in each column.

If a column contains missing values, we need to decide how to handle them before training our model.

Step 5: Find and Remove Duplicate Rows

Duplicate messages can affect our analysis and evaluation.

First, let’s count the duplicate rows.

print("Duplicate rows:", df.duplicated().sum())

Now, let’s display them.

duplicates = df[df.duplicated()]

duplicates

We can remove duplicate rows using the following code.

# Drop duplicates BEFORE splitting
df = df.drop_duplicates().reset_index(drop=True)
df.head(2)

Here:

• drop_duplicates() removes repeated rows.
• reset_index(drop=True) creates a fresh index after removing duplicates.

This helps us work with a cleaner dataset.

Step 6: Check the Distribution of Spam and Ham Messages

Now, let’s check how many spam and normal messages are present in our dataset.

print("\nClass counts:\n", df["label"].value_counts(), sep="")

df["target"] = (df["label"] == "spam").astype(int)

df

The value_counts() function counts the number of messages in each class.

Next, we will visualize the results.

sns.countplot(data=df, x="label")

plt.title("Distribution of Spam and Ham Messages")
plt.xlabel("Message Type")
plt.ylabel("Number of Messages")
plt.show()

The count plot helps us compare the number of spam and ham messages visually.

It also helps us identify whether one class has significantly more examples than the other.

Check the Class Distribution in Percentages

print("\nClass %:\n", (df["label"].value_counts(normalize=True) * 100).round(2), sep="")

Setting normalize=True returns the proportion of each class. Multiplying by 100 converts these proportions into percentages.

This is useful because class imbalance can affect how we evaluate our classification models.

Conclusion

In this first step, we downloaded the SMS Spam Collection dataset, explored its structure, checked for missing values and duplicates, and analyzed the distribution of spam and ham messages.

Our next step is to prepare the text messages for machine learning. We will convert the text into numerical features so that classification algorithms can learn from it.

Step 7: Text Preprocessing

Machine learning models cannot directly understand text messages. We need to clean the text before converting it into numerical features.

%%writefile spam_utils.py
import re
import numpy as np
import pandas as pd

META_COLS = ["length", "n_words", "n_digits", "upper_ratio", "has_currency", "has_url"]

def clean_text(text):
    """Lowercase, replace URLs / currency / numbers with tokens, drop other punctuation."""
    text = str(text).lower()
    text = re.sub(r"http\S+|www\.\S+", " urltoken ", text)
    text = re.sub(r"[£$€]", " currencytoken ", text)
    text = re.sub(r"\d{5,}", " longnumtoken ", text)   # phone numbers, short codes
    text = re.sub(r"\d+", " numtoken ", text)
    text = re.sub(r"[^a-z\s]", " ", text)
    return re.sub(r"\s+", " ", text).strip()

def meta_features(texts):
    """Simple numeric features computed on the RAW message (log-scaled)."""
    s = pd.Series(list(texts)).astype(str)
    feats = pd.DataFrame({
        "length": s.str.len(),
        "n_words": s.str.split().str.len(),
        "n_digits": s.str.count(r"\d"),
        "upper_ratio": s.apply(lambda t: sum(c.isupper() for c in t) / max(len(t), 1)),
        "has_currency": s.str.contains(r"[£$€]").astype(int),
        "has_url": s.str.contains(r"http|www\.", case=False).astype(int),
    })
    return np.log1p(feats.values)

def meta_feature_names(transformer, input_features):
    return np.array(META_COLS, dtype=object)

def clean_text(text):
    """Lowercase, replace URLs / currency / numbers with tokens, drop other punctuation."""
    text = str(text).lower()
    text = re.sub(r"http\S+|www\.\S+", " urltoken ", text)
    text = re.sub(r"[£$€]", " currencytoken ", text)
    text = re.sub(r"\d{5,}", " longnumtoken ", text)   # phone numbers, short codes
    text = re.sub(r"\d+", " numtoken ", text)
    text = re.sub(r"[^a-z\s]", " ", text)
    return re.sub(r"\s+", " ", text).strip()

text = "WIN £5000! Visit https://example.com NOW 12345"

cleaned = clean_text(text)
print(cleaned)

win currencytoken numtoken visit urltoken now longnumtoken

def meta_features(texts):
    s = pd.Series(list(texts)).astype(str)

    feats = pd.DataFrame({
        "length": s.str.len(),
        "n_words": s.str.split().str.len(),
        "n_digits": s.str.count(r"\d"),
        "upper_ratio": s.apply(
            lambda t: sum(c.isupper() for c in t) / max(len(t), 1)
        ),
        "has_currency": s.str.contains(r"[£$€]").astype(int),
        "has_url": s.str.contains(r"http|www\.", case=False).astype(int),
    })

    return np.log1p(feats)  # Keep column names
display(meta_features(texts))

 length n_words n_digits upper_ratio has_currency has_url
0 2.708050 1.386294 1.609438 0.356675 0.693147 0.000000
1 2.995732 1.609438 0.000000 0.051293 0.000000 0.000000
2 3.258097 1.098612 0.000000 0.039221 0.000000 0.693147

names = meta_feature_names(None, None)
print(names)

['length' 'n_words' 'n_digits' 'upper_ratio' 'has_currency' 'has_url']

Save the Helper Functions to a File

%%writefile spam_utils.py

%%writefile is a Colab/Jupyter magic command. It must be the first line of the cell.

• Instead of running the cell, it saves everything below that line into a file called spam_utils.py.
• We can then use from spam_utils import clean_text like any other Python module.
• This matters when we save the model. The saved pipeline remembers that it uses clean_text from spam_utils. If the function lived only inside the notebook, loading the model in a new session would fail.

Imports and Column Names

import re
import numpy as np
import pandas as pd

META_COLS = ["length", "n_words", "n_digits", "upper_ratio", "has_currency", "has_url"]

• re is Python’s regular expression module, used for finding patterns in text.
• META_COLS is the list of names for the extra features we will create. We use it later to label them.

The Cleaning Function

def clean_text(text):
    text = str(text).lower()

• str(text) makes sure the input is a string, even if it is empty or missing.
• .lower() converts everything to lowercase. FREE MONEY NOW becomes free money now.

Replace URLs with a Token

text = re.sub(r"http\S+|www\.\S+", " urltoken ", text)

• re.sub(pattern, replacement, text) finds every match of the pattern and replaces it.
• http\S+ matches http followed by any non-space characters, such as http://spam.com.
• www\.\S+ matches links that start with www..
• The | means “or”.
• Every URL becomes the word urltoken.

We replace the URL instead of deleting it because the presence of a link is a strong spam signal. The exact link does not matter.

Replace Currency Symbols

text = re.sub(r"[£$€]", " currencytoken ", text)

• [£$€] matches any one of these three symbols.
• £1000 becomes currencytoken 1000.

In the original notebook, string.punctuation does not remove £, and it turns www.prize.com into wwwprizecom. Here, both are handled on purpose.

Replace Numbers

text = re.sub(r"\d{5,}", " longnumtoken ", text)
text = re.sub(r"\d+", " numtoken ", text)

• \d means a digit.
• \d{5,} means five or more digits in a row. These are usually phone numbers or short codes, such as 09061701461, and become longnumtoken.
• \d+ means one or more digits. The remaining numbers, such as 1000, become numtoken.
• The order matters. The long numbers must be replaced first, or the second line would turn them into numtoken too.

Remove Everything Else

text = re.sub(r"[^a-z\s]", " ", text)
    return re.sub(r"\s+", " ", text).strip()

• [^a-z\s] means “anything that is not a letter or whitespace”. The ^ inside the brackets means “not”. All remaining punctuation is replaced with a space.
• Our tokens (urltoken, numtoken, and so on) are made of letters, so they survive this step. That is why the replacements come first.
• \s+ matches one or more spaces, which are collapsed into a single space.
• .strip() removes spaces at the start and end.

Example

example = "WINNER!! You have won £1000. Call 09061701461 or visit www.prize.com now"
print(clean_text(example))

Output:

winner you have won currencytoken numtoken call longnumtoken or visit urltoken now

The Extra Features Function

def meta_features(texts):
    s = pd.Series(list(texts)).astype(str)

TF-IDF only looks at words. This function adds a few numbers that describe the message’s shape. It receives the raw message, not the cleaned one. Cleaning lowercases everything, so the uppercase information would already be lost.

• pd.Series(list(texts)) turns the messages into a pandas Series so we can use .str methods.
• .astype(str) makes sure every item is text.

feats = pd.DataFrame({
        "length": s.str.len(),
        "n_words": s.str.split().str.len(),
        "n_digits": s.str.count(r"\d"),
        "upper_ratio": s.apply(lambda t: sum(c.isupper() for c in t) / max(len(t), 1)),
        "has_currency": s.str.contains(r"[£$€]").astype(int),
        "has_url": s.str.contains(r"http|www\.", case=False).astype(int),
    })

• length is the number of characters in the message.
• n_words splits the message on spaces and counts the pieces.
• n_digits counts how many digits appear. Spam often contains many.
• upper_ratio is the share of characters that are uppercase. FREE entry has a higher ratio than see you soon. max(len(t), 1) avoids dividing by zero for an empty message.
• has_currency is 1 if the message contains £, $ or €, otherwise 0. .astype(int) turns True/False into 1/0.
• has_url is 1 if the message contains a link. case=False makes the search ignore capitalization.

return np.log1p(feats.values)

• feats.values converts the table to a NumPy array.
• np.log1p(x) computes log(1 + x). A few very long messages would otherwise dominate the scale. The log squeezes large values together and keeps zeros as zeros.

Feature Names

def meta_feature_names(transformer, input_features):
    return np.array(META_COLS, dtype=object)

• This helper gives our extra features readable names.
• Without it, scikit-learn cannot label them, and Step 11 (top spam and ham words) would fail because get_feature_names_out() would not work.
• It takes two arguments only because scikit-learn passes them in. We ignore both and return META_COLS.
• The names then appear as meta__length, meta__has_url, and so on, next to the TF-IDF words.

import importlib
import spam_utils

importlib.reload(spam_utils)

from spam_utils import clean_text, meta_features, meta_feature_names

example = "WINNER!! You have won £1000. Call 09061701461 or visit www.prize.com now"
print("Raw    :", example)
print("Cleaned:", clean_text(example))

Reload and Test the Helper Functions

This cell does two things: it makes sure Python is using the latest version of spam_utils.py, and it tests clean_text on one example message.

Why we need importlib.reload

import importlib
import spam_utils

importlib.reload(spam_utils)

• import importlib loads Python’s built-in tool for working with imports.
• import spam_utils loads the file we saved with %%writefile.
• Python remembers every module it has already imported. If you import spam_utils, then edit the file and run %%writefile again, Python keeps using the old version from memory.
• importlib.reload(spam_utils) forces Python to read the file again from disk, so the new code is used.

This is common in notebooks, where we edit a helper file and rerun cells without restarting the session. If you never edit spam_utils.py, the reload line does nothing, so it is harmless to keep.

Import the functions

from spam_utils import clean_text, meta_features, meta_feature_names

• This pulls the three functions out of the module so we can call them directly, for example clean_text(...) instead of spam_utils.clean_text(...).
• The order matters. We reload first, then import the names. If we imported the names before reloading, they would still point to the old functions.

Test the cleaning function

example = "WINNER!! You have won £1000. Call 09061701461 or visit www.prize.com now"
print("Raw    :", example)
print("Cleaned:", clean_text(example))

• example is a sample spam-style message with a few things we care about: capital letters, punctuation, a currency symbol, a phone number and a web link.
• The two print lines show the message before and after cleaning, with the labels padded so the outputs line up.

Output:

Raw    : WINNER!! You have won £1000. Call 09061701461 or visit www.prize.com now
Cleaned: winner you have won currencytoken numtoken call longnumtoken or visit urltoken now

Here is what happened to each part:

• WINNER!! became winner: lowercased, with the punctuation removed.
• £1000 became currencytoken numtoken: the symbol and the number are each replaced with a token.
• 09061701461 became longnumtoken: it has five or more digits, so it is treated as a phone number.
• www.prize.com became urltoken: the link is replaced with a token.
• The words you have won call or visit now stayed unchanged.

Why this test matters

Before building the full pipeline, we check that the cleaning does what we expect on one message. If something looks wrong here, it is much easier to fix than after training 15 models.

Check the Cleaned Messages

df[["label", "message"]].head(10)

This displays the first ten rows with their labels and cleaned messages.

It helps us check whether the preprocessing steps worked correctly.

Check Empty and Missing Messages

After cleaning the text, we should check whether any messages are empty or missing.

print("Empty messages:", df["message"].eq("").sum())

print("Missing messages:", df["message"].isnull().sum())

• eq("") identifies messages that contain empty strings.
• isnull() identifies missing values.
• sum() counts how many such messages exist.

An empty message is different from a missing value. An empty message contains no text, while a missing value has no recorded value.

Step 8: Split the Dataset

Before training our models, we need to divide the dataset into training and testing sets.

The training set helps the model learn patterns from the messages. The testing set helps us evaluate how well the model performs on unseen messages.

First, separate the input features and target labels.

from sklearn.model_selection import (train_test_split, 
                                     StratifiedKFold,
                                     cross_validate,
                                     cross_val_predict, 
                                     GridSearchCV)

RANDOM_STATE = 42

X = df["message"]
y = df["target"]

X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=RANDOM_STATE, stratify=y )
print("Train:", len(X_train), "| Test:", len(X_test))
print("Spam share  train: %.3f  test: %.3f" % (y_train.mean(), y_test.mean()))
print("Baseline accuracy (always predict ham): %.4f" % (1 - y_test.mean()))

1. "Spam share: %.3f" is a string enclosed in quotation marks.
2. Python recognizes %.3f as a formatting instruction, not ordinary text.
3. % 0.12345 tells Python to insert this number into the string.
4. .3f means display 3 digits after the decimal point.
5. Python replaces %.3f with 0.123.
6. Output: Spam share: 0.123

Train: 4135 | Test: 1034
Spam share  train: 0.126  test: 0.127
Baseline accuracy (always predict ham): 0.8733

Split the Dataset into Training and Testing Sets

Before training any model, we split the data into two parts. The model learns from one part, and we judge it on the other part, which it has never seen. This tells us how well it will work on new messages.

X = df["message"]
y = df["target"]

Here:

• X contains the text messages that the model will learn from.
• y contains the labels. In this notebook the labels are stored as numbers in the target column: 0 for ham and 1 for spam.

We use the raw messages for X, not cleaned ones. The cleaning happens inside the pipeline later, so each step is fitted only on the training data.

Import the Tools

from sklearn.model_selection import (train_test_split, 
                                     StratifiedKFold,
                                     cross_validate,
                                     cross_val_predict, 
                                     GridSearchCV)

This line imports several tools from sklearn.model_selection:

• train_test_split splits the data. This is the only one we use in this cell.
• StratifiedKFold, cross_validate and cross_val_predict are used later to compare models with cross-validation.
• GridSearchCV is used later to tune hyperparameters.

Set a Random State

RANDOM_STATE = 42

• Splitting involves randomness. Without a fixed seed, you would get a different split every time you run the cell, and your results would change.
• Setting RANDOM_STATE = 42 makes the split repeatable.
• The number 42 has no special meaning. Any fixed number works.
• Saving it in a variable lets us reuse the same value everywhere in the notebook.

Split the Data

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=RANDOM_STATE, stratify=y
)

The function returns four parts:

• X_train contains the messages used to train the model.
• X_test contains the messages used to test the model.
• y_train contains the labels for the training messages.
• y_test contains the labels for the testing messages.

Let’s understand the arguments:

• test_size=0.2 means 20% of the data goes to testing and 80% to training.
• random_state=RANDOM_STATE makes the split repeatable.
• stratify=y keeps the same spam-to-ham ratio in both parts.

Why stratify=y Matters

Spam is only about 12 to 13% of the messages. With a purely random split, the test set could by chance contain far more or far fewer spam messages than the training set, and the evaluation would be unreliable.

With stratify=y, both sets keep nearly the same share of spam. This matters most when one class is much smaller than the other.

Check the Split

print("Train:", len(X_train), "| Test:", len(X_test))
print("Spam share  train: %.3f  test: %.3f" % (y_train.mean(), y_test.mean()))
print("Baseline accuracy (always predict ham): %.4f" % (1 - y_test.mean()))

• The first line prints how many messages ended up in each set. The training set should be about four times larger than the test set.
• The second line prints the share of spam in each set. Because spam is 1 and ham is 0, the mean of the labels equals the fraction of spam. The two numbers should be almost identical, which confirms that stratification worked.
• %.3f formats a number with three decimal places.
• The third line calculates a baseline: the accuracy of a model that predicts “ham” for every message and never detects any spam. Since y_test.mean() is the spam share, 1 - y_test.mean() is the ham share.

Why the Baseline Matters

With about 87% of the messages being ham, a useless model that always says “ham” would already score around 87% accuracy. So a model with 90% accuracy is not as impressive as it sounds.

This is why we also look at precision, recall and F1 score for the spam class. A good model must clearly beat this baseline and also catch spam, not just guess the majority class.

Conclusion

In this step, we converted messages to lowercase, removed surrounding spaces and punctuation, checked empty and missing messages, and divided the data into training and testing sets.

Our text is now cleaner, and the data is ready for the next stage.

In the next step, we will convert text messages into numerical features using techniques such as Bag of Words and TF-IDF. These features will allow us to train and compare different machine learning classification models.

Step 9: Convert Text into Numbers Using TF-IDF

Machine learning models work with numerical data, not raw text. We need to convert our messages into numerical features before training the models.

For this, we will use TF-IDF (Term Frequency–Inverse Document Frequency).

TF-IDF gives a numerical score to words based on their frequency in a message and how common they are across all messages. It helps give more importance to words that are useful for distinguishing messages.

Import the Tools We Need

Before building the models, we import everything we will use in the rest of the notebook. Putting all imports in one cell keeps the code organized, and you can see at a glance which tools the project depends on.

Convert Text into Numbers

from sklearn.feature_extraction.text import TfidfVectorizer

• TfidfVectorizer converts text into numerical features. Machine learning models cannot read words, so every message must become a list of numbers first.
• TF-IDF gives each word a score. Words that appear often in a message but rarely in the rest of the dataset get a higher score, because they help tell messages apart.

Import the Classification Models

We will train and compare several models, so we import them grouped by family.

from sklearn.naive_bayes import MultinomialNB, BernoulliNB, ComplementNB

• Naive Bayes models are fast, simple and often strong for text classification.
• MultinomialNB works with word counts or TF-IDF scores.
• BernoulliNB looks at whether a word is present or absent.
• ComplementNB is designed to work better when one class is much larger than the other, as with our ham and spam messages.

from sklearn.linear_model import LogisticRegression, RidgeClassifier, SGDClassifier

• LogisticRegression is a simple and reliable model that also gives a probability for each prediction.
• RidgeClassifier is a linear model with regularization, which helps prevent overfitting.
• SGDClassifier trains a linear model step by step, so it is fast on large datasets.

from sklearn.svm import LinearSVC, SVC

• Support Vector Machines look for the boundary that separates the two classes with the widest margin.
• LinearSVC uses a straight boundary and is very effective on text data.
• SVC can learn curved boundaries with a kernel, but it is slower.

from sklearn.neighbors import KNeighborsClassifier

• KNeighborsClassifier (KNN) classifies a message by looking at the most similar messages in the training data and taking a vote.

from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import (RandomForestClassifier, ExtraTreesClassifier,
                              AdaBoostClassifier, GradientBoostingClassifier)

• DecisionTreeClassifier makes decisions by asking a series of yes/no questions.
• Ensemble models combine many models to get better results.
• RandomForestClassifier and ExtraTreesClassifier build many trees and average their votes.
• AdaBoostClassifier and GradientBoostingClassifier build models one after another, and each new model tries to fix the mistakes of the previous ones.

from sklearn.neural_network import MLPClassifier

• MLPClassifier is a basic neural network, which can learn more complex patterns.

Import the Evaluation Tools

from sklearn.metrics import (classification_report, confusion_matrix, ConfusionMatrixDisplay,
                             roc_auc_score, average_precision_score, precision_recall_curve,
                             RocCurveDisplay, PrecisionRecallDisplay)

These tools measure how well the models perform:

• classification_report prints precision, recall and F1 score for each class.

1. Precision: Out of all messages the model predicted as spam, how many were actually spam?
2. Recall: Out of all actual spam messages, how many did the model correctly identify?
3. F1 Score: A single score that balances precision and recall to show how well the model identifies spam.

• confusion_matrix counts correct and wrong predictions: how many spam messages were caught, how many were missed, and how many ham messages were wrongly flagged.
• ConfusionMatrixDisplay draws the confusion matrix as a chart.
• roc_auc_score measures how well the model separates spam from ham across all possible thresholds. A score of 1.0 is perfect, and 0.5 is no better than guessing.

ROC-AUC stands for Receiver Operating Characteristic — Area Under the Curve. ROC measures how well a model separates spam from normal messages at different thresholds.
AUC measures the overall ability to distinguish between the two classes; closer to 1 is better.

• average_precision_score summarizes the precision-recall curve in one number (PR-AUC). It is especially useful when spam is much rarer than ham.
• precision_recall_curve calculates precision and recall at different thresholds. We use it later to choose a threshold.
• RocCurveDisplay and PrecisionRecallDisplay draw the ROC curve and the precision-recall curve.

Why We Compare So Many Models

We do not know in advance which model will work best on this dataset. Instead of guessing, we train many of them under the same conditions and compare their scores. The best choice is often a simple model such as Logistic Regression or Naive Bayes, not the most complex one.

Build the Pipeline and Define the Models

In this step we create a function that builds a complete pipeline, and a dictionary with all the models we want to compare.

A pipeline chains several steps into one object. Raw text goes in, a prediction comes out. Every step is fitted only on the training data, which prevents data leakage, and the same steps are applied to any new message later.

Import the Pipeline Tools

This cell uses four extra tools, so make sure they are imported:

from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.preprocessing import FunctionTransformer, MinMaxScaler

• Pipeline runs steps one after another.
• FeatureUnion runs several feature builders side by side and joins their outputs.
• FunctionTransformer turns an ordinary Python function into a pipeline step.
• MinMaxScaler rescales numbers into a fixed range.

The build_pipeline Function

def build_pipeline(clf):
    features = FeatureUnion([
        ("tfidf", TfidfVectorizer(preprocessor=clean_text, ngram_range=(1, 2),
                                  min_df=2, sublinear_tf=True)),
        ("meta", Pipeline([
            ("calc", FunctionTransformer(meta_features, feature_names_out=meta_feature_names)),
            ("scale", MinMaxScaler(clip=True)),   # non-negative, so Naive Bayes works too
        ])),
    ])
    return Pipeline([("features", features), ("clf", clf)])

models = {
    "Multinomial NB": MultinomialNB(),
    "Bernoulli NB": BernoulliNB(),
    "Complement NB": ComplementNB(),
    "Logistic Regression": LogisticRegression(max_iter=1000, class_weight="balanced", random_state=RANDOM_STATE),
    "Ridge Classifier": RidgeClassifier(class_weight="balanced"),
    "SGD Classifier": SGDClassifier(class_weight="balanced", random_state=RANDOM_STATE),
    "Linear SVM": LinearSVC(class_weight="balanced", random_state=RANDOM_STATE),
    "Kernel SVM": SVC(kernel="rbf", class_weight="balanced"),
    "KNN (cosine)": KNeighborsClassifier(n_neighbors=5, metric="cosine", algorithm="brute"),
    "Decision Tree": DecisionTreeClassifier(class_weight="balanced", random_state=RANDOM_STATE),
    "Random Forest": RandomForestClassifier(n_estimators=200, class_weight="balanced", random_state=RANDOM_STATE, n_jobs=-1),
    "Extra Trees": ExtraTreesClassifier(n_estimators=200, class_weight="balanced", random_state=RANDOM_STATE, n_jobs=-1),
    "AdaBoost": AdaBoostClassifier(n_estimators=100, random_state=RANDOM_STATE),
    "Gradient Boosting": GradientBoostingClassifier(n_estimators=100, random_state=RANDOM_STATE),
    "MLP": MLPClassifier(hidden_layer_sizes=(64, 32), max_iter=300, early_stopping=True, random_state=RANDOM_STATE),
}
print(len(models), "models ready")

def build_pipeline(clf):

The function takes one argument, clf (short for classifier). This is the model that goes at the end of the pipeline. Because the model is a parameter, we can reuse the same pipeline with any of our 15 models and compare them fairly.

Feature 1: TF-IDF on the Text

features = FeatureUnion([
    ("tfidf", TfidfVectorizer(preprocessor=clean_text, ngram_range=(1, 2),
                              min_df=2, sublinear_tf=True)),

FeatureUnion takes a list of (name, step) pairs. The first one is the TF-IDF vectorizer. Let’s understand its settings:

• preprocessor=clean_text applies our cleaning function to every message before the text is split into words. Because it is part of the pipeline, new messages are cleaned in exactly the same way.
• ngram_range=(1, 2) uses single words and pairs of neighboring words. For example, free prize becomes free, prize and free prize. Pairs capture phrases that single words miss.
• min_df=2 means the model ignores any word that appears in only one message. It keeps words that appear in at least 2 messages, helping remove rare words and typos.
• sublinear_tf=True replaces the word count with 1 + log(count). A word repeated 10 times is not 10 times more important than a word that appears once.

Feature 2: The Extra Numeric Features

("meta", Pipeline([
        ("calc", FunctionTransformer(meta_features, feature_names_out=meta_feature_names)),
        ("scale", MinMaxScaler(clip=True)),
    ])),
])

The second branch is a small pipeline of its own with two steps:

• calc uses FunctionTransformer to run our meta_features function on the raw message. It creates the numbers for length, word count, digit count, uppercase ratio, and the currency and URL flags. The feature_names_out=meta_feature_names part gives these features readable names.
• scale uses MinMaxScaler to squeeze every value into the range 0 to 1, so these features are on a similar scale to the TF-IDF scores.
• clip=True forces any unseen test value that falls outside the training range to stay between 0 and 1. Without it, a new message could produce a negative number.

The scaler matters for the Naive Bayes models, which cannot accept negative values. That is why the code comment says “non-negative, so Naive Bayes works too”.

FeatureUnion then places the TF-IDF columns and the 6 extra columns side by side into one feature table.

Add the Classifier

return Pipeline([("features", features), ("clf", clf)])

The final pipeline has two named steps:

• features builds all the numbers from the raw text.
• clf is the model that learns from those numbers.

The names matter. Later, when we tune hyperparameters, we refer to settings such as features__tfidf__ngram_range or clf__C, using a double underscore to move down through the steps.

The Models Dictionary

models = {
    "Multinomial NB": MultinomialNB(),
    ...
}

This dictionary stores each model under a readable name. We can loop over it, train every model in the same way and collect the results in one table.

Handling the Class Imbalance

Look at class_weight="balanced" in many of the models:

"Logistic Regression": LogisticRegression(max_iter=1000, class_weight="balanced", random_state=RANDOM_STATE),

Spam is only about 13% of the messages, so a model can score well by mostly predicting ham. With class_weight="balanced", mistakes on the rarer class (spam) are penalized more heavily, so the model pays closer attention to it.

Not every model supports this option. Naive Bayes, AdaBoost, Gradient Boosting, KNN and MLP do not have a class_weight parameter, so they use their default behavior.

Other Settings Worth Knowing

• max_iter=1000 (Logistic Regression) gives the solver enough steps to converge. The default of 100 can trigger a warning on text data.
• random_state=RANDOM_STATE makes models with randomness repeatable.
• LinearSVC uses a straight decision boundary, which works very well for text. SVC(kernel="rbf") can learn curved boundaries but is slower.
• KNeighborsClassifier(n_neighbors=5, metric="cosine", algorithm="brute") compares messages by the angle between their vectors, which suits text better than the default distance. algorithm="brute" is required for cosine distance on sparse data.
• n_estimators=200 (Random Forest and Extra Trees) builds 200 trees. n_jobs=-1 uses all CPU cores.
• n_estimators=100 (AdaBoost and Gradient Boosting) builds 100 models in sequence.
• MLPClassifier(hidden_layer_sizes=(64, 32), ...) creates a neural network with two hidden layers of 64 and 32 neurons. early_stopping=True stops training when the score on a small validation portion stops improving, which helps prevent overfitting.

Check That Everything Is Ready

print(len(models), "models ready")

This prints 15 models ready. Nothing has been trained yet. We have only defined the pipeline recipe and the list of models. In the next step we test each model with cross-validation.

Import the Remaining Tools and Set Notebook Options

This cell imports a few more tools and sets some global options for the notebook.

Import Saving and Version Tools

import joblib
import sklearn

import os, re, time, urllib.request

from sklearn.base import clone
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.preprocessing import FunctionTransformer, MinMaxScaler

N_JOBS = -1          # set to 1 if parallel jobs ever fail
sns.set_theme(style="whitegrid")
pd.set_option("display.max_colwidth", 200)

import joblib
import sklearn

• joblib saves the trained pipeline to a file and loads it back later. We use it at the end of the notebook.
• sklearn is imported as a whole so we can read its version number with sklearn.__version__. We store this version next to the saved model, because a model saved with one version of scikit-learn may not load correctly in a very different version.

Import Standard Python Tools

import os, re, time, urllib.request

This line imports four modules in one go:

• os works with files and folders, for example to create the saved_model folder.
• re is the regular expression module for text patterns. Our clean_text function uses it inside spam_utils.py.
• time measures how long things take. It is not used in the final code, so you can remove it.
• urllib.request downloads the dataset from the internet.

Import Pipeline Tools

from sklearn.base import clone
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.preprocessing import FunctionTransformer, MinMaxScaler

• clone makes a fresh, untrained copy of a model with the same settings. We use it in the threshold step so the original pipeline is not changed.
• Pipeline chains steps one after another.
• FeatureUnion builds several kinds of features side by side and joins them.
• FunctionTransformer turns a normal Python function into a pipeline step.
• MinMaxScaler rescales numbers into the range 0 to 1.

Set the Notebook Options

N_JOBS = -1          # set to 1 if parallel jobs ever fail

• N_JOBS controls how many CPU cores scikit-learn uses for cross-validation and tuning.
• -1 means use all available cores, which makes the notebook faster.
• If you see strange errors related to parallel processing, change it to 1 to run on a single core.
• We store it in a variable so there is one place to change instead of many.

sns.set_theme(style="whitegrid")

• This sets the default style for all Seaborn and Matplotlib charts.
• "whitegrid" gives a white background with light grid lines, which makes bar heights and curves easier to read.
• It only needs to run once. Every chart after this cell uses the style.

pd.set_option("display.max_colwidth", 200)

• By default, pandas cuts long text in a table cell and shows ....
• This option lets each cell show up to 200 characters.
• It is useful when we later print the misclassified messages, since SMS messages can be fairly long.

Make Sure the Earlier Imports Exist

This cell uses sns and pd, so these lines must have run before it:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

Compare the Models with Cross-Validation

In this step we measure how good each of the 15 models is. We use only the training data, and the test set stays untouched.

Why Cross-Validation?

A single train/validation split can be lucky or unlucky. With 5-fold cross-validation, the training data is divided into 5 equal parts (folds). The process is repeated 5 times:

• Train on 4 folds.
• Evaluate on the 1 remaining fold.
• Each fold takes a turn as the evaluation fold.

We then average the 5 scores. This gives a more reliable estimate than one split. Because the pipeline is rebuilt for every fold, the TF-IDF vocabulary is learned only from the training folds, which avoids leakage.

Set Up the Cross-Validation

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=RANDOM_STATE)
scoring = {"accuracy": "accuracy", "precision": "precision",
           "recall": "recall", "f1": "f1", "roc_auc": "roc_auc"}
pretty = {"accuracy": "Accuracy", "precision": "Precision",
          "recall": "Recall", "f1": "F1", "roc_auc": "ROC-AUC"}

rows = []
for name, clf in models.items():
    print(f"CV: {name} ...", end=" ")
    try:
        s = cross_validate(build_pipeline(clf), X_train, y_train, cv=cv,
                           scoring=scoring, n_jobs=N_JOBS)
        row = {"Model": name}
        row.update({pretty[m]: s[f"test_{m}"].mean() for m in scoring})
        row["F1 std"] = s["test_f1"].std()
        row["Fit time (s)"] = s["fit_time"].mean()
        rows.append(row)
        print("done")
    except Exception as e:
        print("FAILED:", e)

cv_df = pd.DataFrame(rows).sort_values("F1", ascending=False).reset_index(drop=True)
cv_df.round(4)

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=RANDOM_STATE)

• n_splits=5 creates 5 folds.
• shuffle=True mixes the data before splitting, in case the file is ordered in some way.
• random_state=RANDOM_STATE makes the folds repeatable.
• Stratified means every fold keeps the same spam-to-ham ratio as the full training data. This matters because spam is only about 13% of the messages.

Choose the Metrics

scoring = {"accuracy": "accuracy", "precision": "precision",
           "recall": "recall", "f1": "f1", "roc_auc": "roc_auc"}

This dictionary tells scikit-learn which scores to calculate. The keys are the names we choose, and the values are scikit-learn’s built-in scorer names.

• Accuracy is the share of all predictions that are correct. It can be misleading here, because always predicting ham already gives about 87%.
• Precision answers: of the messages we flagged as spam, how many were really spam? Low precision means real messages are being sent to the spam folder.
• Recall answers: of all the real spam messages, how many did we catch? Low recall means spam gets through.
• F1 combines precision and recall into one number. It is high only when both are high.
• ROC-AUC measures how well the model ranks spam above ham, regardless of the threshold.

By default, scikit-learn treats class 1 as the positive class. Since we encoded spam as 1, precision, recall and F1 are all calculated for spam, which is exactly what we want.

pretty = {"accuracy": "Accuracy", "precision": "Precision",
          "recall": "Recall", "f1": "F1", "roc_auc": "ROC-AUC"}

This dictionary maps the short names to nicer column titles for the final table.

Loop Over All Models

rows = []
for name, clf in models.items():
    print(f"CV: {name} ...", end=" ")

• rows is an empty list that will collect one result per model.
• models.items() gives each model’s name and object, one at a time.
• The print shows progress. end=" " keeps the cursor on the same line, so done appears next to the model name.

Run Cross-Validation

try:
    s = cross_validate(build_pipeline(clf), X_train, y_train, cv=cv,
                       scoring=scoring, n_jobs=N_JOBS)

• build_pipeline(clf) creates a fresh pipeline with the current model.
• cross_validate trains and evaluates the pipeline on each of the 5 folds, using the raw X_train messages.
• scoring=scoring asks for all five metrics.
• n_jobs=N_JOBS runs the folds in parallel.
• The result s is a dictionary with arrays of 5 values each, for example s["test_f1"] has the F1 score of each fold, and s["fit_time"] has the training time of each fold.

The try block is there because one failing model should not stop the whole comparison.

Save the Results

row = {"Model": name}
row.update({pretty[m]: s[f"test_{m}"].mean() for m in scoring})
row["F1 std"] = s["test_f1"].std()
row["Fit time (s)"] = s["fit_time"].mean()
rows.append(row)
print("done")

• The first line starts a dictionary with the model’s name.
• The second line adds one entry per metric. It uses a dictionary comprehension: for each metric m, it takes the 5 fold scores s["test_m"], averages them with .mean(), and stores the result under the pretty column name.
• F1 std is the standard deviation of the 5 F1 scores. A small value means the model is stable across folds. A large value means its performance depends heavily on which data it sees.
• Fit time (s) is the average training time per fold. This helps us see which models are slow.
• The row is added to the rows list, and done is printed.

Handle Failures

except Exception as e:
    print("FAILED:", e)

If a model raises an error, the loop prints the error message and moves on to the next model. Failed models are simply missing from the final table.

Build the Results Table

cv_df = pd.DataFrame(rows).sort_values("F1", ascending=False).reset_index(drop=True)
cv_df.round(4)

• pd.DataFrame(rows) turns the list of dictionaries into a table with one row per model.
• .sort_values("F1", ascending=False) puts the model with the highest F1 at the top.
• .reset_index(drop=True) renumbers the rows from 0 after sorting.
• cv_df.round(4) displays the table with four decimal places.

How to Read the Table

• Look at F1 first, then check that Precision and Recall are both high and not too far apart.
• Check F1 std. If two models have similar F1, prefer the more stable one.
• Check Fit time (s). A slightly better score is not always worth a much slower model.
• Do not pick the final model from this table alone. In the next step we tune the top candidates, and only then do we use the test set once.

CV: Multinomial NB ... done
CV: Bernoulli NB ... done
CV: Complement NB ... done
CV: Logistic Regression ... done
CV: Ridge Classifier ... done
CV: SGD Classifier ... done
CV: Linear SVM ... done
CV: Kernel SVM ... done
CV: KNN (cosine) ... done
CV: Decision Tree ... done
CV: Random Forest ... done
CV: Extra Trees ... done
CV: AdaBoost ... done
CV: Gradient Boosting ... done
CV: MLP ... done
Model Accuracy Precision Recall F1 ROC-AUC F1 std Fit time (s)
0 Linear SVM 0.9896 0.9764 0.9405 0.9580 0.9936 0.0093 0.4433
1 SGD Classifier 0.9896 0.9801 0.9367 0.9578 0.9923 0.0078 0.6760
2 Kernel SVM 0.9867 0.9646 0.9290 0.9462 0.9938 0.0135 1.7169
3 Ridge Classifier 0.9865 0.9719 0.9195 0.9447 0.9936 0.0109 0.9492
4 Extra Trees 0.9860 0.9816 0.9061 0.9422 0.9935 0.0088 4.3263
5 Complement NB 0.9855 0.9639 0.9195 0.9411 0.9833 0.0099 0.8601
6 MLP 0.9855 0.9837 0.9002 0.9393 0.9855 0.0262 7.1552
7 Random Forest 0.9838 0.9978 0.8735 0.9314 0.9949 0.0126 3.8540
8 Logistic Regression 0.9819 0.9213 0.9367 0.9288 0.9924 0.0196 1.4462
9 KNN (cosine) 0.9814 0.9497 0.9003 0.9240 0.9739 0.0145 0.4390
10 AdaBoost 0.9804 0.9513 0.8907 0.9197 0.9877 0.0138 3.4315
11 Gradient Boosting 0.9797 0.9437 0.8945 0.9177 0.9883 0.0244 6.5104
12 Bernoulli NB 0.9799 1.0000 0.8409 0.9133 0.9943 0.0185 0.7125
13 Multinomial NB 0.9761 0.9863 0.8218 0.8965 0.9833 0.0132 1.3421
14 Decision Tree 0.9732 0.8893 0.9004 0.8946 0.9420 0.0186 0.7293

Plot the Model Comparison

A table is precise, but a chart makes the differences easier to see. This cell draws a horizontal bar chart of the cross-validated F1 score for each model.

Create the Figure

plt.figure(figsize=(10, 6))
sns.barplot(data=cv_df, x="F1", y="Model", color="steelblue")
plt.xlim(0.8, 1.0)
plt.title("Cross-validated F1 (spam class)")
plt.show()

plt.figure(figsize=(10, 6))

• plt.figure() creates a new empty chart.
• figsize=(10, 6) sets its size in inches: 10 wide and 6 tall.
• We need extra height because there are 15 model names on the vertical axis, and a small chart would make the labels overlap.

Draw the Bars

sns.barplot(data=cv_df, x="F1", y="Model", color="steelblue")

• sns.barplot draws a bar for each row of a table.
• data=cv_df is the results table from the previous step.
• x="F1" sets the bar length to the F1 score.
• y="Model" puts the model names on the vertical axis. A horizontal layout is better here because the names are long.
• color="steelblue" uses one color for all bars. The length alone carries the information.

Since cv_df is already sorted by F1 in descending order, the best model appears at the top and the worst at the bottom.

Zoom In on the Axis

plt.xlim(0.8, 1.0)

• plt.xlim(0.8, 1.0) makes the horizontal axis run from 0.8 to 1.0 instead of starting at 0.
• Good models on this dataset all score close to 1. On a full 0 to 1 axis, the bars would look almost identical.
• Zooming in makes small but real differences visible.

Keep two things in mind when you use this:

• A zoomed axis makes small gaps look large. A difference of 0.01 in F1 may be visually dramatic, but it may be within the normal fold-to-fold variation (the F1 std column).
• Any model with an F1 below 0.8 will have its bar cut off or hidden. If a model seems to be missing from the chart, lower the first number, for example plt.xlim(0.5, 1.0).

Add a Title and Show the Chart

plt.title("Cross-validated F1 (spam class)")
plt.show()

• plt.title() adds a title. We say “Cross-validated” so readers know the scores come from 5-fold cross-validation on the training data, not from the test set. We say “spam class” because F1 is calculated for spam, the positive class.
• plt.show() displays the chart.

How to Read the Chart

• Longer bars mean better F1 scores.
• Look for a group of models at the top with nearly equal scores. In that case, the choice between them can depend on other factors, such as training time, stability and whether the model gives probabilities.
• Models far below the rest, such as a single decision tree or KNN, are unlikely to be worth tuning.
• Use this chart to pick the candidates for the next step, hyperparameter tuning.

Tune the Best Models with Grid Search

In the last step we compared 15 models with their default settings. Now we take a few strong candidates and look for their best settings. This is called hyperparameter tuning.

Get Mohammad Atif’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

A hyperparameter is a setting that we choose before training, such as how strongly the model is regularized. The model does not learn it from the data. Different values can noticeably change the results.

What Is Grid Search?

Grid search tries every combination of the values we list, scores each combination with cross-validation, and keeps the best one. Because it uses only the training data, the test set stays untouched.

Settings for the Text Features

tfidf_grid = {
    "features__tfidf__ngram_range": [(1, 1), (1, 2)],
    "features__tfidf__min_df": [1, 2],
    "features__tfidf__sublinear_tf": [True, False],
}

candidates = {
    "Logistic Regression": (LogisticRegression(max_iter=1000, class_weight="balanced", random_state=RANDOM_STATE),
                            {"clf__C": [0.1, 1, 10, 100]}),
    "Linear SVM": (LinearSVC(class_weight="balanced", random_state=RANDOM_STATE),
                   {"clf__C": [0.1, 0.5, 1, 5]}),
    "Complement NB": (ComplementNB(),
                      {"clf__alpha": [0.01, 0.1, 0.5, 1.0]}),
}

searches = {}
for name, (clf, grid) in candidates.items():
    print(f"Tuning {name} ...")
    gs = GridSearchCV(build_pipeline(clf), {**tfidf_grid, **grid},
                      scoring="f1", cv=cv, n_jobs=N_JOBS, refit=True)
    gs.fit(X_train, y_train)
    searches[name] = gs
    print(f"  best CV F1 = {gs.best_score_:.4f}")
    print(f"  best params = {gs.best_params_}\n")

tuned_df = (pd.DataFrame({"Model": list(searches),
                          "Best CV F1": [g.best_score_ for g in searches.values()]})
            .sort_values("Best CV F1", ascending=False).reset_index(drop=True))
best_name = tuned_df.loc[0, "Model"]
best_pipe = searches[best_name].best_estimator_
print("Selected model:", best_name)
tuned_df.round(4)

tfidf_grid = {
    "features__tfidf__ngram_range": [(1, 1), (1, 2)],
    "features__tfidf__min_df": [1, 2],
    "features__tfidf__sublinear_tf": [True, False],
}

This dictionary lists the TF-IDF settings to try. The key names follow the pipeline structure, with a double underscore going one level deeper:

• features is the FeatureUnion step.
• tfidf is the vectorizer inside it.
• ngram_range is the setting we want to change.

The values to try:

• ngram_range: (1, 1) uses single words only. (1, 2) adds pairs of neighboring words.
• min_df: 1 keeps every word. 2 ignores words that appear in only one message.
• sublinear_tf: True dampens repeated words with a log. False uses raw counts.

That makes 2 × 2 × 2 = 8 combinations of text settings.

The Candidate Models

candidates = {
    "Logistic Regression": (LogisticRegression(...), {"clf__C": [0.1, 1, 10, 100]}),
    "Linear SVM": (LinearSVC(...), {"clf__C": [0.1, 0.5, 1, 5]}),
    "Complement NB": (ComplementNB(), {"clf__alpha": [0.01, 0.1, 0.5, 1.0]}),
}

Each entry stores the model and its own settings to try as a pair. The key clf__C refers to the C parameter of the clf step in the pipeline.

• C (Logistic Regression and Linear SVM) controls regularization. A small C means strong regularization, which gives a simpler model that is less likely to overfit. A large C lets the model fit the training data more closely.
• alpha (Complement NB) is a smoothing value. A small alpha trusts the training counts more. A larger alpha smooths them, which helps with rare words.

We chose these three because they are fast, strong on text, and usually near the top of this kind of comparison. If your own table from the previous step shows different top models, replace them here.

Run the Search

searches = {}
for name, (clf, grid) in candidates.items():
    print(f"Tuning {name} ...")

• searches is an empty dictionary where we will keep each finished search.
• for name, (clf, grid) in candidates.items() unpacks each entry into the name, the model and its settings.

gs = GridSearchCV(build_pipeline(clf), {**tfidf_grid, **grid},
                  scoring="f1", cv=cv, n_jobs=N_JOBS, refit=True)
gs.fit(X_train, y_train)

• build_pipeline(clf) creates the full pipeline for this model.
• {**tfidf_grid, **grid} merges the text settings and the model settings into one dictionary. The ** operator unpacks a dictionary into another one.
• scoring="f1" chooses the best combination by F1 score for spam.
• cv=cv reuses the same 5 stratified folds as before.
• n_jobs=N_JOBS runs the work in parallel.
• refit=True means that after finding the best settings, the search retrains the pipeline on the whole training set using those settings. The result is stored as best_estimator_.
• gs.fit(X_train, y_train) runs the search.

For Logistic Regression, there are 8 text combinations × 4 values of C = 32 combinations. With 5 folds, that is 160 training runs, plus one final refit.

Save and Print the Results

searches[name] = gs
print(f"  best CV F1 = {gs.best_score_:.4f}")
print(f"  best params = {gs.best_params_}\n")

• gs.best_score_ is the average cross-validated F1 of the best combination.
• gs.best_params_ is the dictionary of the winning settings.
• We store the finished search in searches so we can use it afterward.

Pick the Winner

tuned_df = (pd.DataFrame({"Model": list(searches),
                          "Best CV F1": [g.best_score_ for g in searches.values()]})
            .sort_values("Best CV F1", ascending=False).reset_index(drop=True))

This builds a small table with each model’s name and its best score, sorted from highest to lowest.

best_name = tuned_df.loc[0, "Model"]
best_pipe = searches[best_name].best_estimator_
print("Selected model:", best_name)
tuned_df.round(4)

• tuned_df.loc[0, "Model"] takes the name in the first row, which is the best model.
• searches[best_name].best_estimator_ gets that model’s pipeline, already retrained on the full training set with the best settings.
• We print the winner and show the table.

output

Tuning Logistic Regression ...
  best CV F1 = 0.9581
  best params = {'clf__C': 100, 'features__tfidf__min_df': 2, 'features__tfidf__ngram_range': (1, 1), 'features__tfidf__sublinear_tf': False}

Tuning Linear SVM ...
  best CV F1 = 0.9597
  best params = {'clf__C': 5, 'features__tfidf__min_df': 1, 'features__tfidf__ngram_range': (1, 1), 'features__tfidf__sublinear_tf': False}

Tuning Complement NB ...
  best CV F1 = 0.9464
  best params = {'clf__alpha': 0.5, 'features__tfidf__min_df': 2, 'features__tfidf__ngram_range': (1, 2), 'features__tfidf__sublinear_tf': True}

Selected model: Linear SVM
Model Best CV F1
0 Linear SVM 0.9597
1 Logistic Regression 0.9581
2 Complement NB 0.9464

Things to Keep in Mind

• The best score here comes from cross-validation on the training data, so it is a fair estimate. It can still be slightly optimistic, because we picked the maximum among many tries.
• If the best value sits at the edge of a list, for example C = 100 is the last value, the true best may lie beyond it. Extend the list and run again.
• Small differences between models are often within the normal variation. If two models are almost tied, you can choose the simpler or faster one.
• The test set has still not been used. In the next step we evaluate best_pipe on it once.

Evaluate the Final Model on the Test Set

So far, every decision (which model, which settings) was made with cross-validation on the training data. Now we evaluate the chosen model on the test set, which it has never seen. This is the honest estimate of how it will behave on new messages.

We do this once. If you go back, change the model after seeing these results and test again, the test set is no longer unseen, and the estimate becomes optimistic.

A Helper to Get Scores

def get_scores(pipe, X):
    """Probability of spam if available, otherwise the decision-function score."""
    if hasattr(pipe, "predict_proba"):
        return pipe.predict_proba(X)[:, 1]
    return pipe.decision_function(X)

y_pred = best_pipe.predict(X_test)
test_scores = get_scores(best_pipe, X_test)

print("Model:", best_name)
print(classification_report(y_test, y_pred, target_names=["ham", "spam"], digits=4))
print("ROC-AUC:", round(roc_auc_score(y_test, test_scores), 4))
print("PR-AUC :", round(average_precision_score(y_test, test_scores), 4))

ROC and precision-recall curves need a score for each message, not just a final label. Different models provide scores in different ways, so this function handles both cases:

• hasattr(pipe, "predict_proba") checks whether the model can output probabilities. Logistic Regression and Naive Bayes can.
• predict_proba(X) returns two columns: the probability of ham and the probability of spam. [:, 1] selects the second column, the spam probability.
• LinearSVC and RidgeClassifier have no probabilities, but they have decision_function. It returns a number for each message: the further above zero, the more confident the model is that the message is spam, and the further below zero, the more confident it is that the message is ham.

Both types of score work for ROC and precision-recall curves, because those curves only need a ranking of the messages.

Make Predictions

y_pred = best_pipe.predict(X_test)
test_scores = get_scores(best_pipe, X_test)

• best_pipe.predict(X_test) gives the final label for each test message: 0 for ham and 1 for spam. We pass the raw messages, and the pipeline cleans them and builds the features automatically.
• test_scores stores the score of each message, which we use for the curves.

Print the Classification Report

print("Model:", best_name)
print(classification_report(y_test, y_pred, target_names=["ham", "spam"], digits=4))

• classification_report compares the true labels with the predictions and prints a table for each class.
• target_names=["ham", "spam"] replaces the labels 0 and 1 with readable names.
• digits=4 shows four decimals.

The table has these columns:

• precision: of the messages predicted as this class, how many were correct.
• recall: of the real messages of this class, how many were found.
• f1-score: the balance of the two.
• support: the number of real messages of that class in the test set.

Pay the most attention to the spam row. The ham row will look very good almost automatically, because ham is the majority class. The report also shows overall accuracy, a macro average (a plain average of the two classes) and a weighted average (weighted by support).

Print ROC-AUC and PR-AUC

print("ROC-AUC:", round(roc_auc_score(y_test, test_scores), 4))
print("PR-AUC :", round(average_precision_score(y_test, test_scores), 4))

• ROC-AUC is the probability that the model gives a random spam message a higher score than a random ham message. 1.0 is perfect and 0.5 is random guessing.
• PR-AUC (average precision) summarizes the precision-recall curve. It is more informative than ROC-AUC when the positive class is rare, because it focuses on how well spam is found without too many false alarms.

Both functions receive the scores, not the labels. They measure the quality of the ranking before any threshold is applied.

Output

Model: Linear SVM
              precision    recall  f1-score   support

         ham     0.9923    0.9989    0.9956       903
        spam     0.9920    0.9466    0.9688       131

    accuracy                         0.9923      1034
   macro avg     0.9921    0.9727    0.9822      1034
weighted avg     0.9923    0.9923    0.9922      1034

ROC-AUC: 0.9952

Draw Three Charts Side by Side

fig, ax = plt.subplots(1, 3, figsize=(17, 4.5))
ConfusionMatrixDisplay.from_predictions(y_test, y_pred, display_labels=["Ham", "Spam"],
                                        cmap="Blues", ax=ax[0])
ax[0].set_title("Confusion matrix")
RocCurveDisplay.from_predictions(y_test, test_scores, ax=ax[1])
ax[1].set_title("ROC curve")
PrecisionRecallDisplay.from_predictions(y_test, test_scores, ax=ax[2])
ax[2].set_title("Precision-Recall curve")
plt.tight_layout()
plt.show()

fig, ax = plt.subplots(1, 3, figsize=(17, 4.5))

• plt.subplots(1, 3) creates one row with three empty plots. The plots are stored in ax, and we refer to them as ax[0], ax[1] and ax[2].
• figsize=(17, 4.5) makes the figure wide enough for three charts.

1. Confusion matrix

ConfusionMatrixDisplay.from_predictions(y_test, y_pred, display_labels=["Ham", "Spam"],
                                        cmap="Blues", ax=ax[0])
ax[0].set_title("Confusion matrix")

This draws a 2 × 2 grid:

• Top left: ham correctly predicted as ham.
• Top right: ham wrongly flagged as spam (false positives). These are real messages that would end up in the spam folder.
• Bottom left: spam that slipped through as ham (false negatives).
• Bottom right: spam correctly caught.

cmap="Blues" colors the cells, with darker blue for bigger numbers. ax=ax[0] places the chart in the first slot.

2. ROC curve

RocCurveDisplay.from_predictions(y_test, test_scores, ax=ax[1])
ax[1].set_title("ROC curve")

• The curve shows the true positive rate (the share of spam caught) against the false positive rate (the share of ham wrongly flagged) at every possible threshold.
• A curve that hugs the top-left corner is excellent. The diagonal line is random guessing.

3. Precision-Recall curve

PrecisionRecallDisplay.from_predictions(y_test, test_scores, ax=ax[2])
ax[2].set_title("Precision-Recall curve")

• The curve shows the trade-off between precision and recall as the threshold changes.
• A curve staying near the top-right corner means the model can catch most spam while keeping few false alarms.
• If you lower the threshold to catch more spam (higher recall), precision usually drops.

plt.tight_layout()
plt.show()

• plt.tight_layout() adjusts the spacing so the titles and labels do not overlap.
• plt.show() displays the figure.

How to Read the Results

• Compare the test F1 for spam with the cross-validated F1 from the tuning step. If the two are close, the model generalizes well. A large drop would suggest overfitting.
• Look at the confusion matrix and ask which mistake is more costly. For a spam filter, a false positive (a real message lost) is usually worse than a missed spam.
• If you want fewer false positives, you do not need a new model. You can raise the decision threshold, which is what the next step does.

Tune the Decision Threshold

A classifier does not output “spam” or “ham” directly. It first produces a score for each message, then compares that score to a threshold. Anything above the threshold is called spam.

• For probabilities, the default threshold is 0.5.
• For decision_function scores, the default threshold is 0.

The default is not always the best choice. For a spam filter, the two mistakes do not cost the same:

• A false positive (a real message sent to spam) can make someone miss something important.
• A false negative (a spam message reaching the inbox) is only annoying.

Raising the threshold makes the model more careful: it flags fewer messages, so precision goes up, and recall goes down. In this step we set a precision target and find the threshold that reaches it.

Choose the Target

TARGET_PRECISION = 0.99

method = "predict_proba" if hasattr(best_pipe, "predict_proba") else "decision_function"
oof = cross_val_predict(clone(best_pipe), X_train, y_train, cv=cv, method=method, n_jobs=N_JOBS)
if oof.ndim == 2:
    oof = oof[:, 1]

prec, rec, thr = precision_recall_curve(y_train, oof)
ok = np.where(prec[:-1] >= TARGET_PRECISION)[0]

if len(ok):
    threshold = thr[ok[0]]
    y_pred_thr = (test_scores >= threshold).astype(int)
    print(f"Chosen threshold: {threshold:.4f} (target precision >= {TARGET_PRECISION})")
    print(classification_report(y_test, y_pred_thr, target_names=["ham", "spam"], digits=4))
else:
    print("Target precision not reachable on the training folds; keep the default threshold.")

TARGET_PRECISION = 0.99

This means: when the filter says “spam”, it should be right at least 99% of the time. You can change the value depending on how much you want to avoid false positives.

Why We Use the Training Data Here

We do not pick the threshold on the test set. If we did, we would tune on the same data we use to report the final result, and the numbers would look better than they really are.

Instead, we get honest scores for the training messages using cross-validation, choose the threshold from those, and then apply it to the test set. The test set is still used only for the final check.

Get Out-of-Fold Scores

method = "predict_proba" if hasattr(best_pipe, "predict_proba") else "decision_function"

This line chooses which kind of score to use, the same way as the get_scores function: probabilities if the model has them, otherwise the decision function.

oof = cross_val_predict(clone(best_pipe), X_train, y_train, cv=cv, method=method, n_jobs=N_JOBS)

• clone(best_pipe) makes a fresh, untrained copy of the pipeline with the same tuned settings.
• cross_val_predict splits the training data into 5 folds. For each fold, it trains on the other 4 and predicts scores for the held-out one.
• The result, oof (out-of-fold), has one score for every training message, and each score comes from a model that did not see that message during training.
• method=method tells it to return scores instead of labels.

if oof.ndim == 2:
    oof = oof[:, 1]

• predict_proba returns two columns, so oof has 2 dimensions. This line keeps only the spam column.
• decision_function already returns one column, so nothing changes.

Calculate Precision at Every Threshold

prec, rec, thr = precision_recall_curve(y_train, oof)

This function tries every useful threshold and returns three arrays:

• prec: the precision at each threshold.
• rec: the recall at each threshold.
• thr: the thresholds, sorted from low to high.

As the threshold goes up, precision generally rises and recall falls.

Find the Lowest Threshold That Reaches the Target

ok = np.where(prec[:-1] >= TARGET_PRECISION)[0]

• prec[:-1] >= TARGET_PRECISION checks which thresholds reach the precision target. The [:-1] is needed because prec has one more value than thr. The extra last value belongs to a final point (precision 1, recall 0) with no threshold, so we drop it to keep the two arrays aligned.
• np.where(...)[0] returns the positions where the condition is true.

if len(ok):
    threshold = thr[ok[0]]

• len(ok) is greater than zero if at least one threshold reached the target.
• ok[0] is the first, and therefore the lowest, threshold that works. The lowest one is the best choice because it keeps recall as high as possible while still meeting the precision target.

Apply It to the Test Set

y_pred_thr = (test_scores >= threshold).astype(int)
print(f"Chosen threshold: {threshold:.4f} (target precision >= {TARGET_PRECISION})")
print(classification_report(y_test, y_pred_thr, target_names=["ham", "spam"], digits=4))

• test_scores >= threshold is True for every test message whose score reaches the threshold. .astype(int) converts True/False to 1/0.
• The code then prints the chosen threshold and a new classification report.

Compare this report with the earlier one. You should see that the spam precision goes up and the spam recall goes down. That is the trade-off, and it is the point of the step.

What If the Target Is Not Reachable?

else:
    print("Target precision not reachable on the training folds; keep the default threshold.")

If no threshold reaches 99% precision on the training folds, the code prints a message and leaves the default threshold in place. In that case, try a lower target such as 0.98.

Chosen threshold: 0.2165 (target precision >= 0.99)
              precision    recall  f1-score   support

         ham     0.9880    1.0000    0.9939       903
        spam     1.0000    0.9160    0.9562       131

    accuracy                         0.9894      1034
   macro avg     0.9940    0.9580    0.9751      1034
weighted avg     0.9895    0.9894    0.9892      1034

Things to Keep in Mind

• The precision on the test set may be a little different from 0.99. The threshold was chosen on training data, and the test set is a different sample. This is expected, and it is an honest result.
• The test set has few spam messages (about 150), so a single mistake changes the numbers noticeably. Do not over-interpret small differences.
• Whether the trade-off is worth it depends on your use case. If missing spam is almost as bad as losing a real message, the default threshold may be fine.
• If you deploy this model, you must also store the chosen threshold next to it, and use it when predicting new messages. Otherwise the saved model will use the default threshold.

Analyze the Model’s Mistakes

Metrics tell us how many mistakes the model makes. They do not tell us why. In this step we read the messages the model got wrong. This is called error analysis, and it often gives more useful ideas for improvement than another round of tuning.

errors = pd.DataFrame({"message": X_test.values, "true": y_test.values, "pred": y_pred})

fp = errors[(errors.true == 0) & (errors.pred == 1)]
fn = errors[(errors.true == 1) & (errors.pred == 0)]

print(f"False positives (ham flagged as spam): {len(fp)}")
display(fp[["message"]])
print(f"\nFalse negatives (spam missed): {len(fn)}")
display(fn[["message"]])

Build a Table of Results

errors = pd.DataFrame({"message": X_test.values, "true": y_test.values, "pred": y_pred})

This creates a table with one row per test message and three columns:

• message is the original text.
• true is the real label (0 for ham, 1 for spam).
• pred is the model’s prediction.

We use .values on X_test and y_test to get plain arrays. These two are pandas Series that keep the index numbers of the original dataset, and the index values would not match y_pred, which is a NumPy array. Using .values on all three makes the rows line up by position.

The messages are the raw ones, before cleaning. That is what we want here, because we want to read what the person actually wrote, with capital letters, links and symbols.

Separate the Two Types of Errors

fp = errors[(errors.true == 0) & (errors.pred == 1)]
fn = errors[(errors.true == 1) & (errors.pred == 0)]

Each line filters the table with two conditions joined by & (and). The parentheses around each condition are required.

• fp holds the false positives: the real label is ham (0), but the model predicted spam (1). These are real messages that would have been sent to the spam folder.
• fn holds the false negatives: the real label is spam (1), but the model predicted ham (0). These are spam messages that got through.

The two types have different costs. For a spam filter, false positives are usually the more serious problem, so read that list first.

Show the Mistakes

print(f"False positives (ham flagged as spam): {len(fp)}")
display(fp[["message"]])
print(f"\nFalse negatives (spam missed): {len(fn)}")
display(fn[["message"]])

• len(fp) and len(fn) count the mistakes of each type. These numbers match the off-diagonal cells of the confusion matrix.
• fp[["message"]] selects only the message column. The double brackets keep the result as a table, so display can show it nicely.
• display() shows a table in Jupyter and Colab. It works in notebooks without an import. In a regular Python script, you would use print() instead.
• \n in the second print adds an empty line before the heading.

We set display.max_colwidth to 200 earlier, so long messages are shown in full instead of being cut off.

What to Look For

When you read the messages, check for patterns:

• False positives are often ham messages that look like spam. For example, a friend sending a phone number, a message with a link, a message mentioning money or a prize, or a message in all capital letters. In that case, the model learned that these features usually mean spam, and the exceptions confuse it.
• False negatives are often spam messages that look like normal conversation. They may be very short, have no link or number, or be written in a friendly tone.
• Unusual language such as slang, abbreviations or typos may not match the words seen during training.
• Possible label mistakes. In this dataset, a few messages may be labeled incorrectly. If you read a “false positive” and it really looks like spam, the model may be right and the label wrong.

How to Use What You Find

Error analysis can lead to concrete improvements:

• Add a new feature that captures a pattern you noticed.
• Change the cleaning function if an important signal is being removed.
• Collect or add more examples of the type the model misses.
• Adjust the threshold if most mistakes are borderline cases.

Keep in Mind

• The list is based on the test set, and the test set is small. A pattern you see in two or three messages may be a coincidence.
• If you change the model because of what you see here, evaluate it again with cross-validation on the training data. Do not tune against the test set, or it will stop being a fair final check.
• If there are no mistakes of one type, the table will be empty. That is a good sign for the model, although it may also mean the test set is too small to expose errors.

See Which Features the Model Relies On

A model that only gives a prediction is a black box. For linear models, we can open the box: each feature has a coefficient that shows how much it pushes the prediction toward spam or ham. In this step we print the features with the strongest influence.

Get the Classifier

clf = best_pipe.named_steps["clf"]
if hasattr(clf, "coef_"):
    names = best_pipe.named_steps["features"].get_feature_names_out()
    coefs = clf.coef_.ravel()
    order = np.argsort(coefs)
    print("Top SPAM indicators:")
    for i in order[::-1][:15]:
        print(f"  {names[i]:<35} {coefs[i]:+.3f}")
    print("\nTop HAM indicators:")
    for i in order[:15]:
        print(f"  {names[i]:<35} {coefs[i]:+.3f}")
else:
    print(f"{type(clf).__name__} has no coefficients; skipping.")

clf = best_pipe.named_steps["clf"]

• best_pipe.named_steps is a dictionary of the pipeline’s steps.
• ["clf"] selects the last step, the trained classifier, without the feature-building part.

Check Whether the Model Has Coefficients

if hasattr(clf, "coef_"):

Only linear models store coef_: Logistic Regression, Linear SVM, Ridge and SGD. Naive Bayes, trees, KNN and neural networks do not. This check makes the code skip the analysis safely when the selected model is one of those. The else branch at the bottom prints a message with the model’s class name instead of crashing.

Get the Feature Names and Coefficients

names = best_pipe.named_steps["features"].get_feature_names_out()
coefs = clf.coef_.ravel()

• get_feature_names_out() returns the name of every column the classifier sees: all the TF-IDF words and word pairs, followed by our 6 extra features. The names have prefixes, such as tfidf__free or meta__has_url. This works because we gave the extra features names with meta_feature_names.
• clf.coef_ has the shape (1, number_of_features). .ravel() flattens it into a simple 1D array, so each name matches one coefficient by position.

Sort the Coefficients

order = np.argsort(coefs)

np.argsort does not return the sorted values. It returns the positions that would sort them, from the smallest coefficient to the largest. We then use those positions to look up both the name and the value.

How to Read a Coefficient

• A positive coefficient pushes the prediction toward spam (class 1). The larger the number, the stronger the push.
• A negative coefficient pushes it toward ham (class 0).
• A value near zero means the feature has little influence.

Print the Top Spam Indicators

print("Top SPAM indicators:")
for i in order[::-1][:15]:
    print(f"  {names[i]:<35} {coefs[i]:+.3f}")

• order[::-1] reverses the order, so the largest coefficients come first.
• [:15] keeps the first 15.
• names[i] and coefs[i] give the name and value for each position i.
• In the f-string, :<35 pads the name to 35 characters, left-aligned, so the numbers line up in a column. :+.3f shows the number with three decimals and always includes the sign.

Print the Top Ham Indicators

print("\nTop HAM indicators:")
for i in order[:15]:
    print(f"  {names[i]:<35} {coefs[i]:+.3f}")

This is the same loop without the reversal. order[:15] takes the 15 most negative coefficients, the strongest signs of ham.

What You Will See

• Spam indicators are typically words and tokens such as call, free, claim, urltoken, longnumtoken and currencytoken, and phrases such as to claim.
• Ham indicators are usually everyday conversational words such as ok, me, going, my and home.
• The extra features, such as meta__has_url or meta__n_digits, appear in the list if they matter. Seeing our own tokens near the top confirms that keeping those signals in clean_text was a good decision.

Cautions

• The list explains how the model behaves. It does not prove that a word causes spam. It shows correlations in this dataset.
• Coefficients can be compared with each other here because TF-IDF scores and our scaled extra features are all in a similar range. If the features had very different scales, a large coefficient would be harder to interpret.
• If two features are strongly related, such as a word and a pair that contains it, the model may split the weight between them, so the ranking is not perfectly clean.
• The dataset is old and comes from a specific region and time. Words that look like strong spam signals here may not work as well on newer messages.

Save the Trained Pipeline

Training takes time, and we do not want to repeat it every time we need a prediction. So we save the final pipeline to a file. Later, we can load it and predict on new messages straight away, in another notebook, a web app or an API.

Create the Folder

os.makedirs("saved_model", exist_ok=True)
joblib.dump({
    "model_name": best_name,
    "pipeline": best_pipe,
    "sklearn_version": sklearn.__version__,
    "labels": {0: "ham", 1: "spam"},
}, "saved_model/spam_pipeline.pkl")

print("Saved:", os.listdir("saved_model"))

Saved: ['spam_pipeline.pkl']

os.makedirs("saved_model", exist_ok=True)

• os.makedirs("saved_model") creates a folder called saved_model in the current working directory.
• exist_ok=True prevents an error if the folder already exists, so you can run the cell more than once.

Save Everything in One File

joblib.dump({
    "model_name": best_name,
    "pipeline": best_pipe,
    "sklearn_version": sklearn.__version__,
    "labels": {0: "ham", 1: "spam"},
}, "saved_model/spam_pipeline.pkl")

joblib.dump(object, filename) writes a Python object to a file. Here the object is a dictionary with four items:

• model_name is the name of the winning model, for example "Logistic Regression". It tells us later what is inside the file.
• pipeline is the complete trained pipeline: the cleaning function, the TF-IDF vocabulary, the extra features with their scaling, and the classifier. This is the important part.
• sklearn_version records the scikit-learn version used for training.
• labels maps the numbers to readable names, so whoever loads the file knows that 1 means spam.

We use a dictionary instead of saving only the pipeline because the extra information travels with the model, and we can add more items later.

Why Save the Whole Pipeline?

In the first version of the notebook, the vectorizer and the model were saved as two separate objects, and the cleaning code lived only in the notebook. A new message could easily be processed differently from the training data.

Now the pipeline contains every step, so one call does everything:

pipe.predict(["Hi, are we meeting tomorrow?"])

Raw text goes in, and a label comes out. There is no chance of forgetting a step or applying it in the wrong order.

Why Save the scikit-learn Version?

Saved models are tied to the library version that created them. If you load the file with a very different version, scikit-learn may show a warning or fail, or give slightly different results. Storing the version lets you check this when loading, and tells you which version to install if there is a problem.

Check the Result

print("Saved:", os.listdir("saved_model"))

os.listdir lists the files in the folder. You should see ['spam_pipeline.pkl'], which confirms that the file was written.

Important Notes

• spam_utils.py must be available when loading. Pickle does not store the code of our functions. It stores only their names, such as spam_utils.clean_text. To load the file in a new session, spam_utils.py must be in the same folder or importable. This is why we put the functions in a separate file instead of defining them in the notebook.
• The threshold is not saved. If you plan to use the custom threshold from the previous step, add it to the dictionary, for example "threshold": threshold, and apply it when predicting. Otherwise the loaded model will use the default threshold. Note that threshold exists only if the target precision was reachable.
• Only load files you trust. Loading a pickle can run code, so never load a .pkl file from an unknown source.
• The .pkl file is for Python. If you need to use the model from another language, you would need a different format, such as ONNX.

Load the Saved Pipeline and Predict on New Messages

This is the last step, and it shows why we saved the pipeline. We load it as if we were in a new notebook or an application, and use it on messages it has never seen.

saved = joblib.load("saved_model/spam_pipeline.pkl")
pipe = saved["pipeline"]
print("Loaded:", saved["model_name"], "| trained with sklearn", saved["sklearn_version"])

def predict_messages(pipe, texts):
    if isinstance(texts, str):
        texts = [texts]
    preds = pipe.predict(texts)      # raw text in, cleaning happens inside the pipeline
    return pd.DataFrame({
        "message": texts,
        "prediction": np.where(preds == 1, "spam", "ham"),
        "score": get_scores(pipe, texts).round(3),
    })

new_messages = [
    "Congratulations! You have won a free prize. Call now!",
    "Hi, are we meeting tomorrow?",
    "URGENT! Claim your cash reward now!",
    "Can you pick up milk on your way home?",
    "You've been selected for a £500 gift card. Visit www.claim-now.com",
]
predict_messages(pipe, new_messages)

Loaded: Linear SVM | trained with sklearn 1.6.1
message prediction score
0 Congratulations! You have won a free prize. Call now! spam 0.216
1 Hi, are we meeting tomorrow? ham -1.654
2 URGENT! Claim your cash reward now! ham -0.641
3 Can you pick up milk on your way home? ham -1.885
4 You've been selected for a £500 gift card. Visit www.claim-now.com spam 1.294

Load the File

saved = joblib.load("saved_model/spam_pipeline.pkl")
pipe = saved["pipeline"]
print("Loaded:", saved["model_name"], "| trained with sklearn", saved["sklearn_version"])

• joblib.load reads the file and returns the same dictionary we saved.
• saved["pipeline"] takes out the trained pipeline, which we call pipe.
• The print shows which model was loaded and which scikit-learn version trained it. If you see a version warning, compare this version with the one installed now (sklearn.__version__).

For this to work in a fresh session, spam_utils.py must be in the same folder, because the pipeline needs clean_text and meta_features.

Write a Prediction Function

def predict_messages(pipe, texts):

We wrap the prediction steps in one function that takes the pipeline and the messages, and returns a clean table. This makes the model easy to reuse.

if isinstance(texts, str):
    texts = [texts]

• isinstance(texts, str) checks whether the input is a single message instead of a list.
• If it is, we put it in a list. This lets the function accept both "Hello" and ["Hello", "Hi"].
• It also avoids a common mistake: scikit-learn would treat a plain string as a sequence of characters and process each letter as a separate message.

preds = pipe.predict(texts)      # raw text in, cleaning happens inside the pipeline

We pass the raw messages directly. The pipeline cleans them, builds the TF-IDF and extra features, and runs the classifier. preds contains 0 or 1 for each message.

Build the Result Table

return pd.DataFrame({
    "message": texts,
    "prediction": np.where(preds == 1, "spam", "ham"),
    "score": get_scores(pipe, texts).round(3),
})

The table has three columns:

• message is the original text.
• prediction is the label in words. np.where(condition, a, b) returns a where the condition is true and b otherwise, so each 1 becomes "spam" and each 0 becomes "ham".
• score is the model’s confidence, from the get_scores function we defined earlier. .round(3) keeps three decimals.

How to Read the Score

The meaning of the score depends on the model:

• For models with probabilities, such as Logistic Regression, it is the probability of spam, between 0 and 1. A score of 0.97 means the model is very sure that the message is spam.
• For models without probabilities, such as Linear SVM, it is the decision function value. Positive means spam and negative means ham, and a larger distance from zero means higher confidence. These numbers are not probabilities and can be larger than 1.

In both cases, higher means more spam-like.

Try It on New Messages

new_messages = [
    "Congratulations! You have won a free prize. Call now!",
    "Hi, are we meeting tomorrow?",
    ...
]
predict_messages(pipe, new_messages)

We wrote five test messages: three that look like spam, one normal conversation and one tricky one. The last line returns the table, and the notebook displays it.

You should see the prize, cash reward and gift card messages marked as spam, and the two normal messages marked as ham.

Things to Keep in Mind

• This function uses the model’s default threshold. If you saved a custom threshold, apply it inside the function, for example by replacing preds with (scores >= threshold).astype(int).
• get_scores must be defined in the notebook before this cell runs. If you copy this function into a separate script, copy get_scores as well.
• A few test messages are not a proper evaluation. They only show that the loading and prediction work. The real evaluation was done on the test set.
• The model was trained on SMS messages from an older dataset. It may be less accurate on modern spam, such as phishing links, messages in other languages or newer scam styles.

Download the Model Files from Colab

Files created in Google Colab live on a temporary virtual machine. When the session ends or the runtime resets, the model file, the helper module and everything else on that machine are deleted. This cell copies the two files we need to your own computer.

Why Two Files?

• saved_model/spam_pipeline.pkl is the trained pipeline.
• spam_utils.py is the helper module with clean_text, meta_features and meta_feature_names.

The .pkl file stores only the names of our functions, not their code. To load the model somewhere else, you need both files. Keep them together in the same folder.

try:
    from google.colab import files
    files.download("saved_model/spam_pipeline.pkl")
    files.download("spam_utils.py")
except ImportError:
    print("Not running in Colab; files are in the working directory.")

Try to Import the Colab Tools

try:
    from google.colab import files

• google.colab is a module that exists only inside Google Colab. It gives access to features such as uploading and downloading files.
• files is the part of that module that handles file transfers.
• The import is inside a try block because the module does not exist on other machines. Outside Colab, the import raises an ImportError.

Download the Files

files.download("saved_model/spam_pipeline.pkl")
files.download("spam_utils.py")

• files.download(path) sends the file to your browser, which saves it in your usual downloads folder.
• We call it twice, once for each file.
• Your browser may ask for permission to download multiple files. If the second file does not arrive, allow multiple downloads for Colab in your browser settings and run the cell again.

Handle Other Environments

except ImportError:
    print("Not running in Colab; files are in the working directory.")

• If you run the notebook locally, in VS Code or in Kaggle, the import fails and this message is printed instead of an error.
• On your own computer, no download is needed. The files are already in the folder where the notebook is running.

This try/except pattern lets the same notebook run both in Colab and elsewhere without changes.

Use the Files Later

To use the model on your computer or on a server:

1. Put spam_pipeline.pkl and spam_utils.py in the same project folder.
2. Install a compatible scikit-learn version. Check the version printed when you loaded the model.
3. Load the model with joblib.load and call predict on raw text.

Alternatives

• Save to Google Drive. If you do not want to download the files each time, mount your Drive and save the model there:

from google.colab import drive
drive.mount("/content/drive")

Then use a path such as /content/drive/MyDrive/spam/spam_pipeline.pkl in joblib.dump. Drive files persist after the session ends.

• Upload to Hugging Face or GitHub if you want to share the model. Remember that a .pkl file can execute code when loaded, so people should only load files from sources they trust.

The model must be saved successfully before you can download it.

Thank you for reading my article! I hope this project helped you understand how to build a spam message classifier using machine learning.

Keep learning, keep practicing, and keep building real-world projects.

Feel free to share your thoughts, feedback, or questions in the comments.

Happy Learning!