A great base model for machine learning, but not a great final model.

Ensemble methods are a fantastic way to capitalize on the benefits of decision trees, while reducing their tendency to overfit. However, they can get pretty complex and can turn into black box models. One of the best things we can do as data scientists and machine learning engineers is make sure we know what’s really going on under the hood when we call that sklearn .fit() method. So that’s what we’re going to do today with Decision Trees.

In this article, we are going to go over:

1. Pros of Decision Trees
2. Cons of Decision Trees
3. Ensemble Methods based on Decision Trees
4. How Decision Trees work: The Decision Tree Algorithm, Splitting (Selection) Criteria

What are the pros of Decision Trees?

Decision Trees are great for a variety of reasons. Let’s take a look!

1. Decision Trees can be used for regression or classification, though they are more popular for classification problems. Generally, if you want to use a decision tree for a regression model, you should use an ensemble method.
2. Decision Trees are non-parametric, which is just a fancy way to say that we aren’t making any assumptions about how our data is distributed and our model’s structure (parameters) will be determined from user input and the observations in our sample, rather than being fixed from the data. Non-parametric models are great when we have a lot of data, but not too much knowledge surrounding the data.
3. Decision Trees are interpretable, which means that after we build the model, we can also make inferences about our data, not just predictions. In sklearn, we can do this using the .feature_importances_ attribute.
4. Decision Trees are fast because they’re simple and “greedy”. While we might not prioritize speed when building a final, deployable model, this can be a huge bonus if we’re just trying to build a model to get an initial understanding of our data (more discussion on how this works below!).
5. Finally, data preprocessing is easier with Decision Trees because we don’t have to scale our data — the splits that occur at each node are worried about one feature at a time! Depending on the algorithm used for determining split, categorical features may not have to be encoded numerically either. However, with scikit-learn’s current CART algorithm (again, more later), you will want to preprocess categorical data with the recommended sklearn.preprocessing.LabelEncoder.

What are the cons of Decision Trees?

As we’ve seen, there are many positives to using Decision Trees…depending on the circumstances. It may not be the best choice if we have a small sample size, and for regression, it may not be the best choice if we think we’ll be predicting target values outside of what our training sample contains. Circumstances asides, Decision Trees, like all models, also have some disadvantages.

1. The most significant disadvantage of Decision Trees is that they are prone to overfitting. Decision Trees overfit because you can end up with a leaf node for every single target value in your training data. In fact, that is the default parameter setting for Decision Tree Classifier / Regressor in sklearn. Though you can tune some hyperparameters like max_features, min_samples_split, min_samples_leaf, and max_depth, it is because of this disadvantage that simple Decision Trees are often traded for their more complex cousins: ensemble methods.
2. Decision Trees are also locally optimized, or greedy, which just means that they don’t think ahead when deciding how to split at any given node. Rather, splits are made to minimize or maximize the chosen splitting (selection) criterion— gini or entropy for classification, MSE or MAE for regression — at that split, which we’ll discuss more later.
3. Because of the greedy nature of splitting, imbalanced classes also pose a major issue for Decision Trees when dealing with classification. At each split, the tree is deciding how to best split up classes into the next two nodes. So when one class has very low representation (the minority class), many of those observations can get lost in the majority class nodes, and then prediction of the minority class will be even less likely than it should, if any nodes predict it at all.

Ensemble Methods Based on Decision Trees

Now that we’ve got a pretty good idea of some of the pros and cons of simple Decision Trees, we’ll talk about what makes them such a good base estimator for ensemble methods. Some of the most popular ensemble methods based on Decision Trees are:

• Random Forest (Regressor / Classifier)
• Extremely Randomized Trees (Regressor / Classifier)
• Bagging (Regressor / Classifier)
• Adaptive Booster (Regressor / Classifier)
• Gradient Boost (Regressor / Classifier)
• XGBoost (Regressor / Classifier)

All of these ensemble methods take a decision tree and then apply either bagging (bootstrap aggregating) or boosting as a way to reduce variance and bias. For a quick overview of ensemble methods and the bagging and boosting techniques, check out these slides I made for my Data Science course at General Assembly:

How Decision Trees Work

It’s hard to talk about how decision trees work without an example. This image was taken from the sklearn Decision Tree documentation and is a great representation of a Decision Tree Classifier on the sklearn Iris dataset. I added the labels in red, blue, and grey for easier interpretation.

We see here that each split is binary, or splits into only 2 child nodes. While this is not a necessity for Decision Trees, many implementations, including sklearn’s, are limited to binary splits because considering anything greater is just too computationally heavy — the tree would never fit. It also doesn’t make a huge difference because binary splits can achieve the same result as a multiway split by simply nesting two binary splits! Due to the complexity of the Decision Tree algorithm, however, the splitting calculations made, when limited to only binary splits, might result in slightly different splits from an algorithm that allowed for more. Again, limiting to binary splits is not a major issue, but simply something to consider. So finally: how does the tree decide where to split??

The Decision Tree Algorithm

The “Decision Tree Algorithm” may sound daunting, but it is simply the math that determines how the tree is built (“simply”…we’ll get into it!). The algorithm currently implemented in sklearn is called “CART” (Classification and Regression Trees), which works for only numerical features, but works with both numerical and categorical targets (regression and classification). At each node, it determines the feature and split threshold of that feature which will yield the “largest information gain” for the model. This “information gain” is measured based on the splitting criteria specified by the user.

If you are so mathematically inclined, here is the formula for information gain, taken from the sklearn documentation (where G = information gain, n = number of data points on left/right side of the threshold, Nm = total number of data points, H = the chosen splitting criteria function, Q = the data at node m, Θ = the feature and threshold being evaluated):

But basically, all you need to know is that “information gain” measures how well a given split splits our data so that the target values of the data points in each child node are most homogenous (classification) or closest to each other (regression).

So how are the feature and split threshold chosen at each node? This is a good question with a somewhat complicated answer! In general terms, the algorithm will scan over every possible threshold split for every feature, calculate the “information gain” for each of these different splits, and then choose the split that yielded the highest information gain (how this is measured depends on the splitting criteria described below). Now, this is more intuitive when the feature is a column of 1s and 0s, or a discrete set of options, so there are few possible splits to consider. However, when we have a continuous numerical feature, it gets more complicated.

The sklearn documentation and hours of Google searching “CART algorithm decision trees how to find split threshold for continuous variables sklearn” failed me in uncovering how split thresholds were determined for continuous variables — was every unique value considered as a splitting point, increasing computational load exponentially? Were the values binned into a set number of bins if there were more unique values than that set number of bins? Or perhaps the algorithm just splits at the mean, median, or right down the middle!

After, again, many hours of searching, this Stackoverflow comment from 2014 finally answered the question: when determining what split thresholds to consider for a continuous variable, the sklearn Decision Tree Algorithm (CART) does NOT bin! Instead, it performs the computationally heavy task of checking every possible split — that is, it sorts all the values for that features, then finds each mean between neighboring values and finds the information gain score for that split (unless the two values are less than 1e-7 apart, an arbitrary small step).

So now that we know what is being evaluated for determining the split, that begs the question of how we decide which feature and threshold to use at each node: splitting criteria!

Splitting (Selection) Criteria

On what feature and what value of that feature a node is split depends on the user-specified criteria: gini or entropy for classification, MSE or MAE for regression. While we could go into great detail about any and all of these, for the purposes of this article, it is most helpful to consider that all of these criteria are measuring how helpful a split is in creating the most homogenous child nodes possible based on their target value.

For classification, gini and entropy use different formulas for evaluating “how many observations of each class would be split into each child node if we were to use this split?” Their respective formulas from the sklearn documentation, again, for those mathematically inclined, are:

Gini

Entropy

On the other hand, for regression problems, MSE (mean squared error) and MAE (mean absolute error) evaluate how close to the mean target value of each child node each target value in that node would be, with MSE weighting larger errors as worse than smaller errors. These are both also common metrics used for loss functions in Linear Regression. The formulas from the sklearn documentation are below:

MSE (mean squared error)

MAE (mean absolute error)

And there you have it — what makes a decision tree, and consequently, what makes up the basis of a lot of our beloved ensemble methods! If you’ve made it this far, thanks for sticking with me, or for following your curiosity, or for your dedication to finishing whatever project you might be working on. I hope this overview of Decision Trees and their underlying workings will be helpful as you continue to sharpen your mind and your data science skills! Happy data processing!

Sources

https://towardsdatascience.com/understanding-decision-tree-classification-with-scikit-learn-2ddf272731bd

https://towardsdatascience.com/understanding-decision-trees-for-classification-python-9663d683c952

https://sebastianraschka.com/faq/docs/parametric_vs_nonparametric.html

https://medium.com/@dataakkadian/what-are-parametric-vs-nonparametric-models-8bfa20726f4d

https://stats.stackexchange.com/questions/294033/why-are-decision-trees-not-computationally-expensive

https://scikit-learn.org/stable/auto_examples/preprocessing/plot_discretization.html

https://stackoverflow.com/questions/25287466/binning-of-continuous-variables-in-sklearn-ensemble-and-trees

https://scikit-learn.org/stable/modules/tree.html

https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html

https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html#sklearn.tree.DecisionTreeRegressor

Decision Trees: Understanding The Basis of Ensemble Methods was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Well, that sounds nice, doesn’t it? I sure think so. But it’s a mouthful. With a goal so abstract and ideal-oriented, how does one continue to craft their story by bringing it to life? For me, the answer was a 3-month Data Science Immersive bootcamp at General Assembly in Washington, D.C. By now, you’re probably wondering: How does data science related to immersion school models? And what the heck is “computational linguistics”?? So let’s dive in!

What is computational linguistics?

Computational linguistics combines the study of linguistics with advanced math, statistics, and programming. We create models of language — how it’s stored, how it’s acquired, how it’s accessed — on computers that rely on heavy mathematical and statistical computations. These models can take the form of decision trees, linear regression, neural networks (which are at their base just complex linear regression with gradient descent) as well as much more complex models I have yet to discover, and can make use of complex libraries and modeling toolkits, such as Facebook’s fairseq(-py) (documentation, GitHub).

The aim of the field is to make use of computational structures to build on the systematic and distributive properties of language to help us better understand human language processing, and perhaps replicate it with AI. In fact, computational linguists are often on Artificial Intelligence (AI) teams, working with speech generation, sound recognition, etc.

One example of a project in this field are the annual Duolingo Shared Task Challenges, such as this year’s challenge to evaluate the frequency of different L1 (native language) translations of L2 (target language) prompts to inform the score and feedback to L2 learners.

If you’re interest in learning more, I’d suggest this page from the Stanford Encyclopedia of Philosophy.

What is cognitive linguistics?

Cognitive linguistics is a subset of cognitive science that focuses on linguistics (helpful, right?).

It’s similar to computational linguistics in that it seeks to answer the same types of questions about language, and is also often a part of AI. However, cognitive linguistics blends more psychology, neuroscience, and philosophy into the mix, and is less heavily connected to heavy stats/math (computation). It’s also in its primacy (like computational linguistics), so it’s still pretty hard to define! There are different opinions on methodology and theoretical framework that are being teased out as we speak — how exciting!

The aim of the field, according to the free introduction to The Oxford Handbook of Cognitive Linguistics (2010) is essentially to understand the structure and organization of natural language, the relationship between syntax and semantics, language how we use it, and the relationship between language and thought. Biiiiig task if you ask me.

Many cognitive linguists work in academia, research and development (R&D), and AI as well.

What is data science?

Data science involves the base programming, math, and stats of computational linguistics, but also extends further into business analytics, data engineering, machine learning engineering, and much more. In my General Assembly Data Science Immersive course, we focus on predictive modeling, a skill essential for computational linguistics (I’d say even more so than natural language processing (NLP), which doesn’t actually seem to be very congruent/aligned with linguistics). Some other topics we cover are data cleaning, Bayesian statistics, NLP, unsupervised learning, SQL, data ethics, and neural networks. Many data science skills can be used in academic research of all types (handling big data, data analysis, etc.). However, data science often extends into data security as well, on top of AI and other specialized tech fields.

One example of a data science problem statement could be, when looking at the Ames Housing dataset, what features of a house are most important in predicting market value? Or maybe, as overall condition increases, how does market value change?

This page from the University of California Berkeley is a great description of yet another widespread, sometimes ambiguous field (that I love).

By now, I’m sure you’re wondering the same question I thought you’d be wondering at the beginning of this article: How do these disciplines all relate to immersion school models? And I’ll tell you my take on it, my grand vision, if you will.

I believe firmly that data-informed research can bring about positive change. People believe numbers, but also believe them more when combined with an understanding of the human experiences they describe. Therefore, I think it is essential to explore the processes of learning a language, and, given that the research says something like what I’d expect, communicate these findings to schools and parents and education policy makers to promote immersion schooling. At the same time, I think a good place to start would be to communicate these findings to teachers of second languages and language textbook writers, and then observe the effectiveness of making changes informed by language learning research in those specific pioneering classrooms.

The Center for Applied Linguistics in Washington, D.C. already does this kind of translational work (connecting research to practice) and would be somewhere I’d love to work in the future.

At the same time, Duolingo does a ton of computational linguistics research on second language acquisition (SLA) based on their ample language learning data (and is another place I’d love to work).

So, how could research change language teaching? These changes could look like: changing the order of introduction of different morphosyntactic features of a language, changing the L1/L2 balance in the classroom, using different activities like task-based form-focused exercises or improv-inspired speaking exercises, or bringing more intentionality to error correction. There are tons of applications and possibilities, but I think the most important would be to focus on encoding language in memory.

As a product of immersion schools, I have learned a language without much concerted effort, been motivated to learn another (with much concerted effort), and traveled around the world, motivated by my desire to use these skills to learn about…life. However, I think my language education from Kindergarten through my last year of college could all be improved (though my Chinese classes were preeeetty effective. I have to say, I love the flipped classroom model). Mainly, I think it has to do with encoding — how can we get the vocabulary and grammar we read in a textbook and hear modeled by our teachers into our subconscious, to use to express ourselves in that language? This is a tough question, but I think with the help of computational linguistics research, we can tackle it.

My experiences learning Spanish and Chinese, along with tutoring adult learners of English and then teaching English to elementary school students in Taiwan, are what got me interested in this question. My long-standing love of math, logic, and music (which I think will also be linked to this research in a variety of ways because of its strong relationship to memory, the musical qualities of prosody, and much more) are what makes me feel like I want to and can answer it…or at least start to answer it!