Decision Tree Algorithm in Machine Learning: A Complete Beginner’s Guide

Introduction

Machine Learning is a branch of Artificial Intelligence that enables computers to learn from data and make predictions or decisions.

There are many machine learning algorithms used for classification and regression problems. One of the most popular and easy-to-understand algorithms is the Decision Tree Algorithm.

A Decision Tree works like a flowchart. It makes decisions by asking a series of questions about the input data until it reaches a final prediction.

In this article, we will understand what a Decision Tree is, how it works, its types, important concepts, advantages, disadvantages, and implementation using Python.

1. What is a Decision Tree Algorithm?

A Decision Tree is a supervised machine learning algorithm used for both classification and regression tasks.

It creates a tree-like structure by dividing the dataset into smaller groups based on different conditions or features.

For example, imagine a system that predicts whether a person will play cricket based on weather conditions.

It may ask questions such as:

• Is the weather sunny?
• Is the temperature high?
• Is the humidity normal?

Based on the answers, the model predicts whether the person will play cricket or not.

A Decision Tree contains three important components:

• Root Node: The starting point of the tree. It represents the complete dataset.
• Decision Node: A node where the data is split based on a condition.
• Leaf Node: The final node that gives the prediction.

2. How Does a Decision Tree Work?

A Decision Tree works by splitting the dataset into smaller subsets using the best available features.

The basic steps are:

1. Start with the complete training dataset.
2. Select the best feature for splitting the data.
3. Divide the dataset based on the selected feature.
4. Repeat the process for each resulting subset.
5. Continue until a stopping condition is reached.
6. Generate the final prediction at the leaf node.

The algorithm selects features that help separate the target classes or reduce prediction error.

3. Important Concepts in Decision Trees

A. Entropy

Entropy measures the impurity or uncertainty in a dataset.

If all records belong to the same class, entropy is zero. If the classes are mixed, entropy is higher.

The formula is:

Where:

• (p_i) represents the proportion of records belonging to class i.
• (n) represents the number of classes.

B. Information Gain

Information Gain measures how much uncertainty is reduced after splitting the dataset using a particular feature.

The formula is:

The Decision Tree generally selects the feature with the highest Information Gain when using this splitting criterion.

C. Gini Impurity

Gini Impurity measures the probability of incorrectly classifying a randomly selected sample if it is labelled according to the class distribution in a node.

The formula is:

A lower Gini value indicates a purer node.

4. Types of Decision Tree Algorithms

Decision Trees are mainly used for two types of machine learning problems.

A. Decision Tree Classification

Decision Tree Classification is used when the target variable contains categories or class labels.

Examples:

• Predicting whether an email is spam or not spam.
• Predicting whether a customer will purchase a product.
• Predicting whether a passenger survived or not.

The output is a class label.

B. Decision Tree Regression

Decision Tree Regression is used when the target variable contains continuous numerical values.

Get Sravya Sirapu’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Examples:

• Predicting house prices.
• Predicting salary.
• Predicting product prices.

The output is a numerical value.

5. What is Overfitting in Decision Trees?

Overfitting occurs when a machine learning model learns the training data too closely, including noise and unnecessary details.

In Decision Trees, a tree with too many branches or excessive depth may perform very well on training data but poorly on unseen testing data.

We can reduce overfitting using:

• max_depth : Limits the maximum depth of the tree.
• min_samples_split : Defines the minimum number of samples required to split a node.
• min_samples_leaf : Defines the minimum number of samples required in a leaf node.
• Pruning : Removes unnecessary branches from the tree.

These techniques help improve the model’s ability to generalize to new data.

6. Implementing Decision Tree Classification Using Python

Let’s implement a Decision Tree Classification model using the Iris dataset available in Scikit-learn.

Step 1: Import Libraries

import pandas as pd

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report

Step 2: Load the Dataset

iris = load_iris()

X = iris.data
y = iris.target
print("Dataset Shape:", X.shape)
print("Target Classes:", iris.target_names)

The Iris dataset contains flower measurements and three flower species.

Step 3: Split the Dataset

We divide the dataset into training and testing data.

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,
    random_state=42
)

• 80% of the data is used for training.
• 20% of the data is used for testing.
• random_state=42 ensures a reproducible split.

Step 4: Create and Train the Model

model = DecisionTreeClassifier(
    criterion="gini",
    random_state=42
)

model.fit(X_train, y_train)

Here, criterion="gini" tells the algorithm to use Gini Impurity to select splits.

Step 5: Make Predictions

y_pred = model.predict(X_test)

print("Predictions:", y_pred)

Step 6: Evaluate the Model

accuracy = accuracy_score(y_test, y_pred)

print("Accuracy:", accuracy)
print("Confusion Matrix:")
print(confusion_matrix(y_test, y_pred))
print("Classification Report:")
print(classification_report(
    y_test,
    y_pred,
    target_names=iris.target_names
))

We use accuracy, confusion matrix, precision, recall, and F1-score to understand the model’s performance.

Step 7: Visualize the Decision Tree

import matplotlib.pyplot as plt
from sklearn.tree import plot_tree

plt.figure(figsize=(16, 10))
plot_tree(
    model,
    feature_names=iris.feature_names,
    class_names=iris.target_names,
    filled=True,
    rounded=True
)
plt.title("Decision Tree Classification")
plt.show()

The visualization shows how the model makes decisions using different features.

7. Advantages of Decision Tree Algorithm

1. Easy to Understand: Its tree-like structure makes predictions easy to explain.
2. Handles Classification and Regression: It can solve both types of problems.
3. Less Data Preprocessing: Feature scaling is generally not required.
4. Handles Numerical and Categorical Features: Decision Trees can work with different feature types, although categorical encoding may be needed in some implementations.
5. Captures Non-Linear Relationships: It can learn complex patterns in data.

8. Disadvantages of Decision Tree Algorithm

1. Overfitting: A deep tree may memorize the training data.
2. Unstable: Small changes in training data can produce a different tree.
3. Greedy Approach: It selects the best split at each step rather than searching for the globally optimal tree.
4. Bias Toward Features: Some splitting criteria can favor features with many possible split points or categories.
5. Limited Generalization: A single decision tree may not perform as well as ensemble methods on some complex datasets.

9. Decision Tree vs Other Machine Learning Algorithms

Each algorithm has its own strengths. The best algorithm depends on the dataset and the problem being solved.

10. Real-World Applications of Decision Trees

Decision Trees are used in several industries:

• Banking: Predicting loan approval.
• Healthcare: Supporting disease classification.
• E-commerce: Predicting customer purchasing behavior.
• Education: Predicting student performance.
• Marketing: Customer segmentation and analysis.
• Insurance: Risk assessment and claim classification.

Conclusion

The Decision Tree Algorithm is one of the most important supervised machine learning algorithms. It uses a tree-like structure to make decisions based on different features.

It is easy to understand, works for both classification and regression problems, and does not generally require feature scaling.

However, overfitting can be a major problem, so techniques such as pruning and controlling tree depth are important.

Understanding Decision Trees provides a strong foundation for learning advanced machine learning algorithms such as Random Forest and Gradient Boosting.

Learning machine learning is not just about knowing algorithms. It is about understanding how they work and applying them to real-world problems.