Decision Tree in Scikit-learn:Learing by Splitting
1.The problem it solves:-
Imagine a school that wants to identify students who may need extra academic support. The school has information about students, such as attendance, study hours, previous examination marks and assignment performance. Teachers can analyse this information manually, but it becomes difficult when there are hundreds or thousands of students.
This is a common problem in data classification. Classification means dividing data into different categories based on certain features. For example, students can be classified as “Pass” or “Fail” based on their academic performance. Similarly, banks may classify loan applications into different risk categories, and email services may classify messages as spam or not spam.
The main challenge is to analyse large amounts of data and identify useful patterns. Traditional manual methods can take more time and may not always give consistent results. Therefore, we need a method that can learn from existing data and make predictions for new information.
Decision Trees are one of the machine-learning techniques used to solve such classification problems. They analyse data by asking a series of simple questions and dividing it into smaller groups. This makes the prediction process easier to understand.
2.What Solutions Existed Before:-
Before the development of machine-learning techniques, people used different methods to solve classification problems. Some common approaches included manual rules, if-then statements and statistical methods such as Logistic Regression. These methods were useful, but they also had certain limitations.
2.1 Manual Rules and If-Then Statements:-
Manual rules are simple conditions created by humans to make decisions. For example, a school may use a rule such as: If a student’s attendance is above 75%, the student is eligible for an examination.
These rules are easy to understand and implement. However, when the amount of data increases, creating and managing many rules becomes difficult. Manual rules also depend on human decisions and may not identify hidden patterns in large datasets.
2.2 Logistic Regression:-
Logistic Regression is a statistical machine-learning algorithm mainly used for classification problems. It predicts the probability of an observation belonging to a particular category.
For example, a bank can use Logistic Regression to estimate whether a loan applicant belongs to a low-risk or high-risk category.
It is useful for many classification tasks and provides probability-based predictions. However, it may not perform well when the relationship between input features and the target is highly complex or non-linear without additional feature engineering.
2.3 Other Existing Methods:-
Other machine-learning methods, such as Support Vector Machines (SVM) and Neural Networks, are also used for classification.
These methods can identify complex patterns in data. However, their working process can be difficult for beginners to understand. Some methods may also require more computational resources and careful parameter tuning.
2.4 Limitations of Existing Solutions:-
The earlier methods have some common limitations:
o Manual rules become difficult to manage when data increases.
o Logistic Regression may struggle with complex non-linear relationships.
o Some advanced models are difficult to interpret.
o Model selection and tuning can require additional effort.
These limitations created a need for methods that can learn patterns from data and make predictions in an understandable way. Decision Trees provide one such approach.
3.Why the New Solution Is Preferred:-
Decision Trees are preferred for many classification problems because they are simple to understand and interpret. Unlike some complex machine-learning algorithms, Decision Trees make decisions by following a sequence of questions. These questions divide the data into smaller groups and help the model reach a final prediction.
3.1 Introduction to Decision Trees
A Decision Tree is a supervised machine-learning algorithm used for classification and regression tasks. It represents decisions in the form of a tree-like structure.
The algorithm learns patterns from training data and creates decision rules based on different features. Each internal node represents a question, each branch represents an answer, and each leaf node represents the final prediction.
3.2 Why Decision Trees Are Useful
Decision Trees offer several advantages:
o Easy to Understand: Their tree-like structure makes the decision process simple to follow.
o Easy to Visualise: The complete decision-making process can be represented through a diagram.
o Less Data Preparation: Decision Trees generally do not require feature scaling.
o Handles Different Features: They can work with numerical and categorical features when the data is appropriately prepared.
o Useful for Classification: They can classify new observations into different categories.
o Feature-Based Decisions: They select features that help divide the data into useful groups.
3.3 Comparison with Earlier Solutions
Decision Trees have some advantages over traditional methods. The following table compares different approaches.
Comparison Table
3.4 Trade-offs and Limitations of Decision Trees
Although Decision Trees are useful, they also have some limitations. A tree with too many branches may become very complex and memorise the training data instead of learning general patterns. This problem is known as overfitting.
Scikit-learn provides parameters such as max_depth, min_samples_split and min_samples_leaf to control the size of a tree. These parameters help reduce unnecessary splitting and improve the model’s ability to work with new data.
Therefore, Decision Trees are preferred when interpretability and simple decision-making are important. However, their performance should always be tested on unseen data before using them for real-world predictions.
4. Working Flow: How It Actually Works:-
A Decision Tree works by dividing a dataset into smaller groups based on different conditions. It follows a step-by-step process in which the algorithm asks questions about the input features and makes decisions based on their answers.
The main purpose of this process is to create groups that contain similar data points. The algorithm continues splitting the data until it reaches a stopping condition. Finally, it uses the learned decision rules to predict the class of new data.
4.1 Overview of Decision Tree Structure
A Decision Tree has a tree-like structure consisting of nodes and branches. Each part has a specific role in the decision-making process.
Main components of a Decision Tree:
1. Root Node:-
The root node is the starting point of a Decision Tree. It contains the complete training dataset and represents the first decision or question.
2. Decision Node:-
A decision node represents a condition or question based on a feature. It divides the data into smaller groups.
3. Branch:-
A branch represents the outcome of a decision. It connects one node to another and shows the direction of the decision.
4. Leaf Node:-
A leaf node is the final node of the tree. It represents the final predicted class or output.
4.2 Splitting Criteria
Splitting is the process of dividing a dataset into smaller groups based on selected features. A Decision Tree uses different criteria to select the best split.
1. Gini Impurity
Gini Impurity is a measure used to check how mixed the classes are in a dataset. A lower Gini value means that the data contains more similar classes.
Formula:
Gini = 1 − Σ(pᵢ)²
Here, pᵢ represents the proportion of each class in a node.
2. Entropy
Entropy measures the uncertainty or impurity present in a dataset. Lower entropy means less uncertainty, while higher entropy means more mixed classes.
Formula:
Entropy = −Σ pᵢ log₂(pᵢ)
3. Information Gain
Information Gain measures how much uncertainty is reduced after splitting the data. A Decision Tree prefers a split that provides higher Information Gain.
Formula:
Information Gain = Parent Entropy − Weighted Child Entropy
4.3 Step-by-Step Working Process
The working of a Decision Tree can be explained through the following steps:
1. Data Collection: First, a labelled dataset is collected for training the model.
2. Data Preparation: The input features and target variable are separated. The data is prepared in a suitable format.
3. Selecting the Best Feature: The algorithm examines available features and selects a suitable feature for splitting.
4. Splitting the Data: The dataset is divided into smaller groups based on the selected condition.
5. Repeating the Process: The same process is repeated for the newly created groups until a stopping condition is reached.
6. Creating Leaf Nodes: When splitting stops, the final groups become leaf nodes containing the predicted class.
7. Making Predictions: New data is passed through the decision rules, and the model predicts the final output.
4.4 Decision Tree Working Flow Diagram:-
Caption: Working Flow of a Decision Tree in Scikit-learn.
4.5 Decision Tree Structure Diagram
Caption: Basic Structure of a Decision Tree showing Root Node, Decision Nodes, Branches and Leaf Nodes.
4.6 Small Worked Example:-
Consider a simple example of eight students. We want to predict whether a student will pass or fail based on study hours.
The dataset contains the following information:
The Decision Tree can select a condition such as:
Are study hours greater than 2.75?
· If the answer is No, the prediction is Fail.
· If the answer is Yes, the prediction is Pass.
Get Yuktipatle’s stories in your inbox
Join Medium for free to get updates from this writer.
In this simple example, the two groups are separated perfectly.
4.7 Gini Impurity Before and After Splitting:-
Before splitting, the dataset contains four Pass and four Fail students.
Gini Impurity:
Gini = 1 − (4/8)² − (4/8)²
Gini = 1 − 0.25 − 0.25
Gini = 0.50
After splitting:
· First group contains only Fail students.
· Second group contains only Pass students.
The Gini Impurity of both groups becomes 0 because each group contains only one class.
Caption: Gini Impurity Before and After Splitting.
4.8 Python Concepts and Data Structures Used:-
Scikit-learn uses different Python concepts and data structures to implement Decision Trees.
· Class: DecisionTreeClassifier is a class provided by Scikit-learn.
· Object: An instance of DecisionTreeClassifier represents the model.
· Arrays: Input features and target labels can be stored in arrays.
· Methods: fit() is used to train the model, while predict() is used to make predictions.
· Tree Structure: The trained model stores information about nodes, branches and decision rules.
These concepts help the model learn from data and generate predictions.
4.9 Practical Implementation of Decision Tree Using Scikit-learn
Scikit-learn provides the DecisionTreeClassifier class to implement Decision Tree algorithms in Python. In this example, we will create a simple model that predicts whether a student will Pass or Fail based on study hours.
Step 1: Import Required Libraries
First, import the required libraries.
from sklearn.tree import DecisionTreeClassifier, plot_tree
import matplotlib.pyplot as plt
Step 2: Create a Dataset
We create a small dataset containing students’ study hours and their examination results.
# Study hours of students
X = [[1], [1.5], [2], [2.5],
[3], [3.5], [4], [5]]
# Results: 0 = Fail, 1 = Pass
y = [0, 0, 0, 0, 1, 1, 1, 1]
Here, X represents the input feature (study hours), while y represents the target variable (student results).
Step 3: Create and Train the Model
Now, create a Decision Tree classifier and train it using the dataset.
model = DecisionTreeClassifier(
criterion=’gini’,
max_depth=2,
random_state=42
)
model.fit(X, y)
The fit() method allows the model to learn patterns from the training data.
Step 4: Make Predictions
Now, predict the result of a new student who studies for 3.5 hours.
new_student = [[3.5]]
prediction = model.predict(new_student)
if prediction[0] == 1:
print(“Student Result: Pass”)
else:
print(“Student Result: Fail”)
Output:
Student Result: Pass
Step 5: Visualize the Decision Tree
Scikit-learn also provides the plot_tree() function to visualize the trained Decision Tree.
plt.figure(figsize=(10, 6))
plot_tree(
model,
feature_names=[‘Study Hours’],
class_names=[‘Fail’, ‘Pass’],
filled=True
)
plt.show()
This generates a tree diagram showing the decision conditions, branches and final predictions.
Explanation of the Code:
1. DecisionTreeClassifier(): Creates a Decision Tree classification model.
2. criterion=’gini’: Uses Gini Impurity to select the best split.
3. max_depth=2: Limits the maximum depth of the tree.
4. fit(): Trains the model using the given dataset.
5. predict(): Predicts the result for new input data.
6. plot_tree(): Visualizes the trained Decision Tree.
Conclusion of Implementation:
This practical example demonstrates how Scikit-learn can be used to create, train and visualize a Decision Tree model. It shows how a machine-learning algorithm learns patterns from existing data and uses them to make predictions for new observations.
Caption: Practical Implementation of Decision Tree Classification Using Python and Scikit-learn.
4.10 Time Complexity (Big-O):-
Time complexity describes how the execution time of an algorithm changes as the amount of input data increases.
For Decision Trees:
· Training: A commonly used approximate complexity for balanced trees is O(n × m × log n), where n represents the number of samples and m represents the number of features.
· Prediction: Prediction generally depends on the depth of the tree because the model follows a path from the root node to a leaf node.
The actual time may vary depending on the dataset and tree structure.
4.11 Overfitting and Pruning:-
Overfitting occurs when a Decision Tree becomes too complex and learns the training data too closely. As a result, it may not perform well on new data.
Scikit-learn provides different parameters to control tree growth:
· max_depth: Controls the maximum depth of the tree.
· min_samples_split: Specifies the minimum number of samples required to split a node.
· min_samples_leaf: Specifies the minimum number of samples required in a leaf node.
· ccp_alpha: Controls pruning by removing unnecessary branches.
Pruning helps make the tree simpler and can improve its ability to work with unseen data.
5. Real-World Use Cases:-
Decision Trees are used in different fields to analyse data, identify patterns and make predictions. They help organisations divide information into different categories and support decision-making. Some important real-world applications are explained below.
5.1 Banking and Credit Risk Assessment:-
Banks and financial institutions use decision-tree models to analyse customer information and assess credit risk. These models can help classify loan applicants into different risk categories based on factors such as income, loan history and financial information.
A research study by Jozef Zurada and Chiou Peng Lam examined decision-tree techniques for credit scoring using a German consumer-loan dataset. The study explored how decision trees could classify customers and generate understandable decision rules.
5.2 Credit Card Risk Analysis:-
Decision Trees can also be used in credit card risk management. A case study published by Indus Insights describes how decision-tree modelling was used for a major global credit card issuer.
The model analysed transaction-related information to identify customer groups with a higher likelihood of default. This helped the organisation identify high-risk customer segments and support better risk-management decisions.
5.3 Customer Relationship Management:-
Decision Trees are also useful in customer relationship management (CRM). A research case study involving a car dealership used a decision-tree model to generate recommended customer lists for maintenance services.
The model helped frontline technicians identify relevant customer groups and improve their work efficiency. This example shows how decision trees can support customer analysis and service planning.
5.4 Student Daily-Life Example:-
Consider a college that wants to identify students who may need additional academic support. The college has information about attendance, study hours, assignment completion and previous examination marks.
A Decision Tree can analyse this information and learn patterns from previous student records. Based on these patterns, it can classify students into categories such as “Needs Support” or “Performing Well”.
This can help teachers identify students who may benefit from extra guidance. However, the prediction should only support teachers’ decisions and should not replace their judgement.
5.5 When Decision Trees May Not Be a Good Fit:-
Decision Trees are not suitable for every problem. They may perform poorly when the dataset contains too much noise or when the decision boundaries are very complex. A very deep tree can also suffer from overfitting, which means it may perform well on training data but poorly on new data.
In such situations, other machine-learning algorithms may provide better results. Therefore, the model should always be tested before using it for real-world predictions.
6. Conclusion and References:-
6.1 Conclusion:-
Decision Trees are useful machine learning algorithms that make predictions by splitting data into smaller groups based on selected conditions. Scikit-learn makes it easier to create, train, and use Decision Tree models in Python. Although Decision Trees are easy to understand and interpret, they may suffer from overfitting if they become too complex. Overall, they are a useful choice for classification problems where understanding the decision-making process is important.
6.2 References
Ø Scikit-learn Developers. Decision Trees — Scikit-learn User Guide. [Primary Source]
Ø Zurada, J. and Lam, C. P. Decision Tree Models for Credit Scoring. Research Paper.
Ø Indus Insights. Using Transactions-Based Risk Modeling to Reduce Losses. Case Study.
