ML Guide 02 — Decision Trees

Photo by Patrick Hendry on Unsplash

1. Impurity
2. Building a tree
3. Practice

Recommended Video

• Decision and Classification Trees, Clearly Explained!!!
• StatQuest: Decision Trees, Part 2 — Feature Selection and Missing Data
• Regression Trees, Clearly Explained!!!
• Machine Learning Fundamentals: Bias and Variance

What is a decision tree?

Is a supervised learning, can be used to solve both regression and classification problems.
Decision trees use the tree representation to solve the problem in which each leaf node corresponds to a class label and attributes are represented on the internal node of the tree. We can represent any boolean function on discrete attributes using decision trees.

With Regression

With Classification

When would the tree stop growing?

Don’t build very deep trees in order to avoid overfitting.

How to select a feature and a threshold?

• Choose the decision that will split the data in the equally size subsets
• Choose the decision that will lower the global error
• Need to choose a metric

1. Impurity

• We need to look and measure the quality of a node
• Always look for a low impurity (needs to be pure a node = same value)
• If the results are more uniform it equals low impurity, it equals low deep.

1.1 How to Measure impurity in

1.1.1 Classification Tree

In a classification tree, if its continues splitting into more nodes, if there are a lots of nodes it equals high impurity

• EX: {1,1,0,1,1} is pure than {0,1,0,1,1} “The first is more uniform”
• Percentage of majority class
  {1,1,0,1,1} is pure class 1 at 80% while {0,1,0,1,1} is pure class 1 at 60%
• Entropy
• Gini Index

1.1.2 Regression tree

In a regression tree if it continues to split if the cost is too high, we use MSE (Metric of Cost) to handle it.

1.2 Gini Index

It’s a metric to measure how often a randomly chosen element would be incorrectly identified. It means an attribute with lower Gini Index equals low impurity, that’s what we’re looking for.

p(c|n) is probability of class c at node n(total posibilities).

1.2.1 Example

A node contains samples from 3 classes. Calculate Gini Index for {0,0,1,1,1,2,2,2,2,2}

Get Tσrres’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Class 0: p(0) = 2/10 = 0.2 -> 0.⁰²²=0.04
Class 1: p(1) = 0.3 -> 0.09
Class 2: p(2) = 0.5 -> 0.25
GINI = 1 — (0.04 +0.09+0.25) = 0.62 / 0 = pure, 1 = impure, 0.5 equally distributed.

NBCL = total = 10 “number of classes”

Max Value = 1- 1/NBCL min value = 0

1.3 Entropy (or Log Loss as measure of impurity)

1.3.1 Example

Anode contains samples from 3 classes. Calculate the Entropy for {0,0,1,1,1,2,2,2,2,2}

Class 0: p(0) = 2/10 = 0.2
Class 1: p(1) = 0.3
Class 2: p(2) = 0.5

ENTROPY= — [0.2*log2(0.2) + 0.3*log2(0.3) + 0.5*log2(0.5)] = 1.4

Max Value = 3* log (nbCL)/nbCL

2. Building Tree

We build all the possibilities of the tree and selecting the best ones we are going to find the bst solution, but we cannot do it, it takes so much time
So, we apply “greedy algorithm” to build a tree

• Best decision are taken locally, at each split (node)
• Compare the split candidate by a metric
• Is a viable solution, not the best one
• Be intuitive

Solution for a boolean tree

2.1 Pruning

During the learning process we apply pruning to our tree in order to match the constraints, and after the learning ,it will reduce overfitting.

2.2 Underfit and overfitting

In the real world, the dataset present will never be clean and perfect. It means each dataset contains impurities, noisy data, outliers, missing data, or imbalanced data. Due to these impurities, different problems occur that affect the accuracy and the performance of the model. One of such problems is Overfitting.

An overfitted model doesn’t perform accurately with the test/unseen dataset and can’t generalise well.

Overfitting occurs when the model fits more data than required(is complex), and it tries to capture each and every datapoint fed to it. Hence it starts capturing noise and inaccurate data from the dataset, which degrades the performance of the model.

3. Practice

It is must to follow the exercises from the previous section.

Function predict

Once the model is trained, we can use predict() to predict an output value, on the basis of input values. We call it from an existing instance of a machine learning model that’s already been trained

• model_instance.predict(x_text);
• Model_instance: (Linear regression, SVM, decision trees ,…)
• Predict: the predict method
• X_test: the features of the test data (input needs to be 2D numpy format)

Train data

X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.4)

Basic Tree algorithm in python and plotA

from sklearn import tree

myTree = tree.DecisionTreeClassifier()
myTree.fit(X_train,y_train)
ypred = myTree.predict(X_test)
tree.plot_tree(myTree)

print("Accuracy of a test classification", accuracy_score(y_test,ypred))


ConfusionMatrixDisplay.from_predictions(y_test,ypred)

___

Accuracy of a test classification 0.9333333333333333
<sklearn.metrics._plot.confusion_matrix.ConfusionMatrixDisplay at 0x7feb6a0b38