Decision Tree

Project Overview and Objectives

This project demonstrates the use of a Decision Tree Classifier, a powerful and interpretable supervised machine learning algorithm used for classification tasks. Using Python and the scikit-learn library, a decision tree model is trained on a small dataset and evaluated using a confusion matrix.

The objectives of this project are:

• To understand how decision trees make decisions
• To learn how rule-based classification works
• To implement a decision tree using Python
• To evaluate classification results using a confusion matrix
• To explain the code step by step, line by line

This project emphasizes conceptual clarity by using a simple dataset that clearly illustrates how decision trees split data.

Motivation and Real-World Relevance

Decision trees mimic human decision-making.

Get KVoid’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

We use them constantly:

• If temperature > 38 → fever
• If income > threshold → loan approved
• If email contains keywords → spam

Because of this structure, decision trees are widely used in:

• Medical diagnosis systems
• Credit approval models
• Customer segmentation
• Fraud detection
• Rule-based expert systems

Their biggest strength is interpretability — you can explain why a decision was made.

Basic Theory of Decision Trees

A decision tree works by:

• Splitting data based on feature values
• Asking a sequence of if–else questions
• Reaching a final decision at a leaf node

Each split is chosen to maximize data purity, commonly using:

• Gini Impurity
• Entropy (Information Gain)

For classification:

• The tree continues splitting until classes are well separated
• Final predictions are made by majority class at leaf nodes

Decision trees can easily overfit, which is why depth and splitting rules matter in real applications.

Python Concepts and Libraries Used

This project uses the following tools:

• NumPy — numerical data representation
• scikit-learn — machine learning framework
• DecisionTreeClassifier — decision tree algorithm
• confusion_matrix — evaluation of classification results

Step-by-Step Code Explanation (Line by Line)

Importing Required Libraries

import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import confusion_matrix

Explanation:

• numpy is used to create numerical datasets
• DecisionTreeClassifier implements the decision tree algorithm
• confusion_matrix evaluates prediction performance

Creating the Sample Dataset

X = np.array([[1],[2],[3],[4],[5]])
y = np.array([0, 0, 0, 1, 1])

Here:

• X represents input feature values
• Each value is shaped as a 2D array (required by scikit-learn)
• y represents class labels
• This creates a binary classification problem

Initializing and Training the Decision Tree

model = DecisionTreeClassifier()
model.fit(X, y)

What happens during training:

• The model analyzes feature values
• It finds optimal split points
• Decision rules are created automatically
• The tree learns how to separate class 0 and class 1

Unlike KNN, decision trees learn rules during training.

DecisionTreeClassifier

Making Predictions

y_pred = model.predict(X)

Explanation:

• The trained tree applies learned rules
• Predictions are generated for each input
• Each data point follows a path from root to leaf

Evaluating Using Confusion Matrix

cm = confusion_matrix(y, y_pred)
print(cm)

This step:

• Compares actual labels with predicted labels
• Counts correct and incorrect predictions
• Produces a confusion matrix showing:
• True positives
• True negatives
• False positives
• False negatives

Since predictions are made on training data, the matrix may show perfect accuracy — an example of overfitting, which is important to understand.

Output

Key Learning Outcomes

• Decision trees use rule-based splitting
• They are intuitive and easy to interpret
• Training builds decision rules, not distances
• Confusion matrices reveal prediction behaviour
• Evaluating on training data can hide overfitting

Conclusion

This project demonstrates how a decision tree classifier learns decision rules from data and applies them to classify inputs. By combining a simple dataset with confusion matrix evaluation, it clearly shows how predictions are made and assessed.

Decision trees are powerful because they balance performance and interpretability, making them a cornerstone of many real-world machine learning systems. Understanding them provides a strong foundation for advanced models like Random Forests and Gradient Boosted Trees.