1. Load and understand the Dataset

The first step in any machine learning project is to truly understand your data. This goes beyond just glancing at a table of numbers; it involves thinking critically about what each column represents and the story behind it.

Key points to keep in mind:

1. Semantic meaning of columns

• Ask yourself: What does this column actually measure?
• Some numeric-looking columns may be categorical in nature, and treating them incorrectly can mislead your model. Some examples are:
  - ZIP Codes (numbers represent locations, not quantities)
  - Phone numbers (unique identifiers, no arithmetic meaning)
  - Customer ID/ Employee ID(identifiers, not continuous values)

2. Check the values and distributions

• Look at ranges, unusual values, or anomalies.
• Identify potential errors, inconsistencies, or placeholders (like zeros or blanks) that might need special handling.

3. Relationships between columns

• Don’t assume a correlation shown in visualizations always indicates a meaningful relationship.
• Consider domain knowledge: do these features logically influence each other or the target?

4. Data collection context

• Understand how the data was gathered: surveys, sensors, logs, or manual entry.
• Knowing this helps you spot potential biases or limitations in the dataset.

5. Identify redundancy or irrelevant columns

• Some columns may be duplicates or proxies for others.
• Removing irrelevant features reduces noise and can improve model performance.

6. Think critically about categorical labels

• Are the categories meaningful and consistent?
• Could some categories be combined, or do some represent too few samples to be useful?

💡 Bottom line: Understanding your data is not just a mechanical inspection; it’s about thinking like a detective, combining domain knowledge with intuition, and questioning every column. This foundational step guides every subsequent preprocessing decision and ensures that your model learns meaningful patterns rather than spurious correlations.

2. Check Data Types

Not all data is created equal. Some columns may appear numeric at first glance but are actually stored as text or object types.

A classic example is the TotalCharges column in the Telco Churn dataset.

While it represents numeric values, it is stored as an object, which allows empty strings to appear for certain rows. These empty or misformatted entries are clearly undesirable, as they can lead to errors or missing values when training a model.

You can see that I have handled this here by dropping the rows that contain empty strings for TotalCharges. I can do that here, as there are only a few rows with this discrepancy in the big dataset, which contains about 7043 rows in total.

# Check unique non-numeric entries
print(df["TotalCharges"].loc[df["TotalCharges"].str.strip() == ""].shape)
# Outputs (11, )

# Convert to numeric (force errors to NaN)
df["TotalCharges"] = pd.to_numeric(df["TotalCharges"], errors="coerce")

# Now check missing values
print(df["TotalCharges"].isnull().sum())
# Outputs 11

# Drop rows since < 1%, a very minimal portion of the dataset is affected
df = df.dropna(subset=["TotalCharges"])

Understanding the true type of each feature is essential because it determines how you handle it during preprocessing. Numeric columns require proper conversion before scaling, while categorical or textual columns need encoding. Similarly, empty or invalid entries in numeric columns must be addressed through cleaning or imputation to prevent issues downstream.

The best practice is to carefully inspect column types and the unique values they contain. This early step ensures that each feature is interpreted correctly and prevents subtle issues from propagating into your model. By taking the time to understand the data types, you lay a strong foundation for all subsequent preprocessing steps.

3. Handle Missing Values

Missing data is one of the most common challenges in machine learning and, if left unaddressed, can lead to biased or inaccurate models. Understanding why data is missing is the first step in deciding how to handle it.

Some reasons that can lead to missing data are:

• Non-response by people. People skip survey questions (e.g., income, age).
• Data entry errors. Typos, accidental blanks, or incomplete forms.
• Sensor failures. Device malfunctions lead to missing readings.
• Software bugs. Data not saved properly due to crashes or sync issues.
• Optional fields. Dataset design allows certain values to be left empty.
• Sampling differences. Some variables are only collected for a subset of subjects.

Note that the reasons involve human, system, and data collection methods as factors.

Sometimes values are missing completely at random, other times there is a pattern related to the feature itself or to the target variable. Ignoring missing entries or treating them incorrectly can distort the relationships in your dataset and negatively impact model performance.

There are multiple strategies to deal with missing values, each suitable for different scenarios.

One simple approach is to drop rows or columns that have too many missing values. This is effective when the proportion of missing data is small or when an entire column is mostly empty, making it unreliable for modeling. However, dropping data is not always ideal, especially if the missing information is substantial or carries meaningful insights.

Note the figure below. It shows the shape of the default Titanic Dataset.

Then we check the columns will missing/null values.

age and deck have many rows with empty/null values. So we drop them.

The result: we lost ~80% of the data points.
It goes to show that we should not drop arbitrarily. In this case, we could have judged the deckcolumn’s relevance and dropped the column itself, or made some imputation (filled up with values) in the column.

Imputation is a more nuanced approach, which involves filling in missing values based on some reasonable assumption.

Simple imputation strategies use basic statistics like the mean, median, or mode of the feature. These are easy to implement and often work well for numeric features with a roughly symmetric distribution or categorical features with a dominant category.

Group-based imputation considers patterns within subgroups of your data — for example, filling in missing values based on the average within a particular category or demographic. This method preserves relationships between features and can lead to more accurate imputation than global statistics.

Target-based imputation is another sophisticated approach where missing values are predicted using other features, effectively treating imputation as a mini supervised learning problem. This can be powerful but requires caution, as it can unintentionally introduce data leakage if the target variable from the test set is used during imputation.

Another valuable technique related to handling numeric features is binning, which involves grouping continuous values into discrete intervals or categories. Binning can simplify complex distributions, reduce the effect of outliers, and sometimes improve model interpretability. For example, ages could be binned into ranges like 0–18, 19–35, 36–50, and 51+, or income could be grouped into quartiles. Binning is particularly useful when the precise numeric value is less important than the range or category to which it belongs.

Ultimately, the goal is to handle missing data thoughtfully and deliberately. Each strategy — whether dropping, simple imputation, group-based imputation, target-based imputation, or binning — comes with trade-offs. The key is to ensure that the chosen method preserves meaningful patterns in the data without introducing bias or leaking information from the test set. Proper handling of missing values and thoughtful binning lay a strong foundation for all subsequent preprocessing and modeling steps.

4. Prevent Data Leakage

Data leakage is a subtle but serious issue in machine learning that can drastically undermine the reliability of your models. In simple terms, data leakage occurs when information from outside the training set inadvertently influences the model during training. This means that the model ends up learning from data it is not supposed to have access to, giving it an unfair advantage. While the model may appear to perform exceptionally well during evaluation, its apparent success is often misleading because it has already “seen” aspects of the test data. In real-world scenarios, where new data is unseen, the model may perform far worse than expected.

A classic example of data leakage is when you are predicting customer churn, and one of the features you include is the total amount billed, which may have been calculated after the observation period. If your model has access to this post-event information during training, it can easily “guess” the churn outcome, but this knowledge would not be available when predicting churn for new customers. Another subtle form of leakage can happen during preprocessing: if you calculate scaling parameters, impute missing values, or encode categories using the entire dataset — including the test data — you unintentionally provide the model with insights about the unseen data.

Data leakage is particularly dangerous because it can give a false sense of confidence in your model’s accuracy. A model that appears to achieve perfect or near-perfect performance due to leakage will likely fail in production. This can lead to poor business decisions, wasted resources, and a lack of trust in machine learning outputs. It is therefore critical to identify potential sources of leakage before training and evaluation.

The most effective way to prevent data leakage is to split your dataset into training and testing sets at the very beginning of your workflow. Once the split is done, all preprocessing, feature engineering, and parameter fitting should be carried out exclusively on the training data. Only after these transformations have been learned from the training set should you apply the same procedures to the test set, using the same parameters. This ensures that the test set remains a true representation of unseen data, allowing your evaluation metrics to accurately reflect the model’s ability to generalize.

In summary, understanding and preventing data leakage is crucial for building trustworthy machine learning models. By carefully separating training and test data and strictly applying preprocessing learned from the training set, you ensure that your model’s performance is genuine and that it can be expected to perform reliably on new, real-world data.

5. Handle Outliers

Outliers are extreme values that deviate significantly from the rest of the data. They are often the result of measurement errors, data entry mistakes, or rare events, and if left unaddressed, they can distort your model’s understanding of patterns. Models that are sensitive to extreme values, such as linear regression or distance-based algorithms, can be heavily influenced by outliers, leading to poor predictions and reduced generalization performance. Therefore, detecting and appropriately handling outliers is a crucial step in preprocessing.

One of the most effective ways to identify outliers is through visualization. Visual tools allow you to intuitively spot unusual points and understand the distribution of your data. Among these, the box plot is a particularly powerful method. Box plots display the median, quartiles, and potential outliers in a dataset, giving a clear view of the spread and skewness of your data. Points that lie beyond the whiskers of the box plot are typically considered outliers and warrant further investigation.

Once outliers are identified, the next step is deciding how to handle them. There are generally two approaches: clipping and removal. Clipping involves setting extreme values to a threshold, effectively limiting their influence without losing the data entirely. This is useful when outliers are genuine but disproportionately large or small compared to the rest of the data. Removal, on the other hand, involves deleting rows that contain extreme values, which can be appropriate when outliers are clearly erroneous or irrelevant. The choice between clipping and removal depends on the context, the nature of the data, and the specific goals of the analysis.

Handling outliers thoughtfully can make your model more robust and prevent it from being unduly influenced by rare events. By combining visualization techniques like box plots with deliberate decisions about clipping or removal, you ensure that your models learn meaningful patterns from the majority of the data while minimizing the impact of anomalous points. This step not only improves model performance but also contributes to more reliable and interpretable results.

6. Exploratory Data Analysis (EDA)

Exploratory Data Analysis, or EDA, is a critical step in the machine learning workflow that involves examining your dataset to understand its underlying characteristics. The goal of EDA is to uncover patterns, detect anomalies, and identify relationships among features, which in turn guides preprocessing, feature engineering, and modeling decisions. While it is not always mandatory, performing EDA thoughtfully can save significant time and effort later by revealing potential issues before training a model.

EDA can be approached at several levels. Univariate analysis focuses on individual features, examining their distributions, central tendencies, and variability. This helps in understanding whether a feature is skewed, contains outliers, or has an unexpected spread of values. Bivariate analysis goes a step further by exploring relationships between two variables, often between features and the target variable. This can help identify predictive relationships or correlations that might be important for modeling. Multivariate analysis considers interactions among multiple features simultaneously, which is particularly useful for complex datasets where feature combinations may influence the target variable in ways not apparent when looking at features in isolation.

Performing EDA allows you to spot potential problems early, such as highly imbalanced classes, correlated features, or unexpected data distributions. It also helps in making informed decisions about transformations, encoding, or feature selection. In essence, EDA is a form of detective work: by systematically exploring your data, you build a deeper understanding of its structure, patterns, and quirks, laying a strong foundation for all subsequent preprocessing and modeling steps.

7. Check Assumptions and Multicollinearity

Certain machine learning models, particularly parametric models like linear regression, rely on specific assumptions about the underlying data. One key assumption is that the features follow a roughly normal distribution, which helps ensure that the model can estimate relationships accurately and that its predictions are reliable. While not all models require normally distributed data, understanding the distribution of your features is an important step in preprocessing, as transformations or scaling may be necessary to meet these assumptions.

Another critical consideration is multicollinearity, which occurs when two or more features are highly correlated with each other. High multicollinearity can distort model coefficients, reduce interpretability, and make the model unstable, as small changes in the data can lead to large fluctuations in the estimated relationships. Checking for multicollinearity ensures that each feature provides independent information to the model, improving both the stability and interpretability of the results. Common methods for detecting multicollinearity include examining correlation matrices or calculating variance inflation factors (VIFs), which highlight features that may need to be removed or combined.

By assessing assumptions and multicollinearity, you not only improve the performance of models like linear regression but also gain confidence that your data is structured in a way that allows meaningful patterns to be learned. Addressing these considerations early in the preprocessing pipeline prevents problems downstream and sets the stage for more robust, interpretable, and reliable models.

8. Scaling and Normalization

Many machine learning algorithms are sensitive to the scale of input features, meaning that features with larger ranges can disproportionately influence the model’s learning process. For example, a feature representing income in thousands might dominate over a feature representing age in tens if both are fed into the model without adjustment. To address this, scaling is often applied to bring numeric features to a common range or variance. This ensures that each feature contributes more evenly to the learning process, preventing any single feature from overwhelming the others.

In addition to scaling, normalization is another important preprocessing step. Normalization transforms data so that features contribute equally to the model, often by adjusting them to a common scale or converting them into probabilities. This is particularly important for algorithms that rely on distance calculations, such as k-nearest neighbors, clustering methods, and support vector machines, as well as for models that use gradient-based optimization, like logistic regression and neural networks.

By carefully scaling and normalizing your features, you ensure that your model can learn efficiently and that the resulting predictions are balanced and meaningful. Neglecting this step can lead to models that converge slowly, become biased toward certain features, or fail to capture the true patterns in the data. Ultimately, scaling and normalization are essential steps in preparing your dataset for robust and reliable machine learning.

Conclusion

Data preprocessing is the backbone of successful machine learning. From understanding your dataset and handling missing values, to preventing data leakage, managing outliers, and scaling features, each step ensures your model learns meaningful patterns rather than noise.

Remember: a clean, well-prepared dataset is often more valuable than a complex model. Treat preprocessing as a fundamental part of your ML workflow, and your models will thank you.

Large Language Models (LLMs) like GPT-4 are incredibly powerful, but they have a fixed knowledge cutoff. This means they do not have information beyond a certain date, and they lack access to domain-specific knowledge unless explicitly trained on it.

One way to provide an LLM with new knowledge is retraining, but this is expensive and impractical for most use cases. Instead, Retrieval-Augmented Generation (RAG) dynamically retrieves relevant context from external sources, making responses more accurate, domain-specific, and context-aware without modifying the LLM itself.

How RAG Works

1. The user asks a question.
2. An information retrieval mechanism searches for relevant information.
3. The retrieved information is merged with the user’s question.
4. The AI generates a response using the augmented query.
5. The final answer is delivered to the user.

How to Build a RAG Chatbot

Collecting Data

1. The chatbot requires a domain-specific knowledge base (e.g., legal documents, medical reports, financial records).
2. These documents can come from various sources: PDFs, websites, databases, or structured files like CSV/JSON.
3. The system processes and cleans the documents to make them easily searchable.

Creating the Knowledge Base

1. Break documents into smaller chunks for better searchability.
2. Convert each chunk into an embedding (numerical representation).
3. Store the embeddings in a vector database (e.g., Pinecone, FAISS, Weaviate).

Retrieving Relevant Information

1. Convert the user query into an embedding.
2. Search the vector database for similar embeddings.
3. Retrieve the most relevant text chunks as context.

Augmenting the User Query

1. The retrieved information is merged with the user’s original question.
2. This ensures the AI has additional knowledge before generating an answer.

Generating the Response

1. The enhanced query is sent to an LLM (e.g., OpenAI’s GPT-4).
2. The model processes both the original query and the retrieved knowledge before crafting a response.

Delivering the Response

1. The chatbot presents a more accurate and fact-based answer to the user.
2. The response can be displayed in a chatbot interface, web app, or voice assistant.

Implementing a RAG Chatbot in Python

We will build a simple RAG chatbot using:

1. Pinecone → Vector Database (stores & retrieves embeddings)
2. OpenAI API → Generates responses using a language model

from dotenv import load_dotenv
import os
from pinecone import Pinecone
from openai import OpenAI
from pdfminer.high_level import extract_text
from sentence_transformers import SentenceTransformer
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Load environment variables from a .env file
# These variables contain sensitive information like API keys needed to connect with external services.
load_dotenv()

# Initialize a pre-trained model that converts text into a numerical format (embedding)
# This is necessary for comparing text similarities using vector search.
embedding_model = SentenceTransformer('all-MiniLM-L6-v2')

# Connect to Pinecone, a specialized database designed for storing and retrieving vector embeddings.
# The connection is established using an API key stored in the environment variables.
pc = Pinecone(api_key=os.getenv('PINECONE_API_KEY'))

# Access a specific Pinecone index where all the stored embeddings will be saved and retrieved.
# An index is like a structured database table optimized for searching similar vectors.
index = pc.Index(os.getenv('PINECONE_INDEX'))

# Initialize the OpenAI client, which allows us to send queries to OpenAI's models for generating responses.
openaiClient = OpenAI()

def get_embedding(text):
    """
    Converts the input text into a high-dimensional vector representation.
    This numerical representation allows us to measure similarities between different pieces of text.
    :param text: The input text to be converted.
    :return: A numerical vector representing the meaning of the input text.
    """
    return embedding_model.encode(text)

def process_pdf(file_path):
    """
    Extracts raw text from a PDF file, splits it into smaller meaningful chunks, 
    converts them into embeddings, and stores these embeddings in the Pinecone vector database.
    :param file_path: The location of the PDF file to process.
    """
    # Extract textual content from the given PDF file
    text = extract_text(file_path)
    
    # Split the extracted text into smaller chunks for better indexing
    # The `chunk_size` determines how large each segment is (500 characters per chunk)
    # `chunk_overlap` ensures slight overlap (50 characters) to maintain context continuity.
    text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
    chunks = text_splitter.split_text(text)

    # Process each chunk separately: Convert it into a numerical vector and store it in Pinecone.
    for i, chunk in enumerate(chunks):
        embedding = get_embedding(chunk)  # Generate the vector representation for the chunk.
        index.upsert(vectors=[{"id": str(i), "values": embedding, "metadata": {"text": chunk}}])

def query_rag_system(query_text):
    """
    Uses a query to find the most relevant text chunk from the Pinecone database,
    then sends this context to OpenAI to generate an appropriate response.
    :param query_text: The user's question or input.
    :return: AI-generated response based on relevant retrieved information.
    """
    # Convert the query into an embedding vector so we can find similar stored embeddings.
    query_embedding = get_embedding(query_text)
    
    # Search Pinecone for the most relevant text chunk
    # Parameters:
    # - vector: The embedding of the query text (must be a list of numerical values).
    # - top_k: The number of most similar results to retrieve (1 means fetch the closest match).
    # - include_metadata: Ensures the stored metadata (text chunk) is retrieved with the match.
    results = index.query(vector=query_embedding.tolist(), top_k=1, include_metadata=True)
    
    # Extract the best-matching text chunk from Pinecone's response.
    context = results["matches"][0].get("metadata", {}).get("text", "")
    
    # Construct a well-formatted prompt using the retrieved context.
    # This ensures OpenAI generates responses based on relevant information.
    prompt = f"Context: {context}\n\nQuestion: {query_text}\nAnswer:"
    
    # Send the query to OpenAI along with the retrieved context, asking it to generate an answer.
    # Parameters:
    # - model: The specific AI model to use (e.g., 'gpt-4o-mini').
    # - messages: A list of messages forming the conversation context. The last message is the user query.
    response = openaiClient.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}]
    )
    
    # Extract and return the generated response in a clean format.
    return response.choices[0].message.content.strip()

# Example Usage:
# Step 1: Process a PDF file and store its content in Pinecone as embeddings.
process_pdf("Dinosaurs_in_India.pdf")

You can serve this system with an API built using Flask or Python. This is what I did and upon querying using a simple frontend I got the following results.

Create a new folder named htmlToPDF (or anything you want). CD into it. Then, run the following to initialize a new npm project with default options:

npm init -y

Now, let’s install Puppeteer:

npm install --save puppeteer

Create a new file named index.js and add the following snippet:

const puppeteer = require('puppeteer');
const fs = require('fs');
const path = require('path');

The necessary modules have been imported here.

Read the HTML file next:

    const htmlFilePath = path.join(__dirname, 'index.html');
    let html = fs.readFileSync(htmlFilePath, 'utf8');

The absolute path of index.html is being read into the htmlFilePath variable. We’re using Nodejs’s ‘path’ module to join the path to the current working directory given by ‘__dirname’ with the actual filename to get the absolute path.

Then we’re using the ‘fs’ module to synchronously read the HTML file into the html variable.

Now, to one of the most important parts, i.e., rendering images stored locally. For that we need to use the image’s base64 form. If we directly use the path of the image, it is not displayed.

“It’s a security block to protect from malicious websites to access local images”, as yujiosaka commented on the github issue regarding the same.

Below we are replacing the ‘src’ of the <img> that we want to be displayed in the PDF with it’s base64 string.

First, we read the image using the ‘fs’ module. fs.readFileSync() here returns a buffer of the contents of ‘image.png’ which is then converted to a base64-encoded string using the toString(‘base64’) method.

Here, process.cwd() returns the path to the current working directory. Also note the double slashes. The reason for putting the additional slash is that to use a backslash character in a string literal in JavaScript, we need to escape it by using another backslash

html = html.replace('image.png', `data:image/png;base64,${fs.readFileSync(`${process.cwd()}\\image.png`).toString('base64')}`);

Now the actual conversion part arrives:

const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.setContent(html);

// generate PDF from HTML
const pdf = await page.pdf({ 
  format: 'Letter',
  path: 'my_pdf.pdf'
});

// close the browser instance
await browser.close();

const browser = await puppeteer.launch();:

This line launches a new instance of the Puppeteer browser using the puppeteer.launch() method. The launch() method returns a Promise that resolves to a Browser object, which represents a running instance of the Chromium browser. The await keyword is used to wait for the Promise to resolve before proceeding to the next line of code.

const page = await browser.newPage();

This line creates a new page in the Chromium browser instance using the browser.newPage() method. The newPage() method returns a Promise that resolves to a Page object, which represents a single tab in the browser.

await page.setContent(html);

This line sets the content of the page to the HTML code contained in the html variable. The setContent() method is an asynchronous method that takes a string of HTML code as its argument and returns a Promise that resolves when the HTML is successfully loaded into the page.

const pdf = await page.pdf({ format: 'Letter', path: 'my_pdf.pdf' });:

This line generates a PDF document from the HTML content of the page using the page.pdf() method and writes it to the path specified, which would be the current directory in this case. The pdf() method takes an options object as its argument, which can include options such as the format of the PDF document (e.g., "Letter", "A4", etc.), the orientation of the document (e.g., "portrait" or "landscape"), and other settings. The pdf() method returns a Promise that resolves to a Buffer containing the PDF document data.

In summary, the code launches a new instance of the Chromium browser, creates a new page in the browser, sets the content of the page to the HTML code contained in the html variable, generates a PDF document from the HTML content of the page, and assigns the resulting PDF Buffer to the pdf variable. And finally, closes the browser instance.

Ultimately, index.js will look like:

// import the necessary modules
const puppeteer = require('puppeteer');
const fs = require('fs');
const path = require('path');

// read the HTML file
const htmlFilePath = path.join(__dirname, 'index.html');
let html = fs.readFileSync(htmlFilePath, 'utf8');

const convertToPDF = async (html) => {

  // replace local image src with it's base64 string
  html = html.replace('image.png', `data:image/png;base64,${fs.readFileSync(`${process.cwd()}\\image.png`).toString('base64')}`);

  // launch browser instance 
  const browser = await puppeteer.launch();

  // create a new page/tab
  const page = await browser.newPage();

  // set the content of the new page
  await page.setContent(html);

  // Generate PDF from HTML
  const pdf = await page.pdf({ 
    format: 'Letter',
    path: 'my_pdf.pdf'
  });

  // close the browser instance
  await browser.close();
}

convertToPDF(html);

I’ve put the code in an ‘async’ function as the ‘await’ keyword can only be used inside of an ‘async’ function.

So, we’re done here. Thanks for the read.

Your feedback will be appreciated.