Hyperparameter Tuning Part1: Introduction.

The difference between parameters and Hyperparameters, how to find and set them, as well as some tips and tricks for prioritizing your efforts.

Outlines.

• Parameters VS Hyperparameters
• Why to Tune the Hyperparameters
• Where to Find Parameters?
• Where to Find Hyperparameters?
• The Hyperparameter Importance
• How to Find hyperparameters that matter?
• Conflicting Hyperparameter Choices
• Automating Hyperparameter Choice
• Learning Curve.

Introduction

Building powerful ML models depends heavily on the set of hyperparameters used. In ML, a model is represented by the learnt parameters. However, the process of training a model involves choosing the optimal hyperparameters that the learning algorithm will use to find out the optimal parameters that correctly map the input features (independent variables) to the labels or targets (dependent variable) such that you achieve some form of intelligence.

But with increasingly complex models with lots of options, how do you efficiently find the best settings for your particular problem?

Along this series of blogs, we will gain practical experience in using some common methodologies for automated hyperparameter tuning, including Grid Search, Random Search and advanced optimization methodologies e.g., Bayesian and Genetic algorithms. As well as we will learn some best practice tips & tricks for choosing which hyperparameters to tune and what values to set & build learning curves to analyze your hyperparameter choices.

Hyperparameters VS Parameters.

Today algorithms are getting more and more complex, and so the number of hyperparameters to choose from increases. It becomes increasingly important to learn how to efficiently find optimal combinations, as this search will likely take up a large portion of your time.
Often, it’s quite easy to simply run Scikit Learn functions on the default settings or perhaps code from a tutorial or book without really digging under the hood. However, what lies underneath is of vital importance to good model building. You may be surprised what you find!

What are Parameters?

Parameters are internal to the model. That is, they are learned or estimated purely from the data during training as the algorithm used tries to learn the mapping between the input features and the labels or targets.

Model training typically starts with parameters being initialized to some values (e.g., zeros, uniform distribution, Glorot Initialization and He Initialization). As training progresses the initial values are updated using an optimization algorithm (e.g. gradient descent). The learning algorithm is continuously updating the parameter values as learning progress but hyperparameter values set by the model designer remain unchanged.

So crucially, we can’t set the parameters. But instead, the ML algorithm discovers them through undertaking its steps.

We pass both the feature values in a Metrix along with the labels corresponding to each instance to the machine learning algorithm to learn the model Parameters to find out patterns to map between features and labels, which yield a generalized function to be able to capable of predicting values or classes on new input data.

What are Hyperparameters?

The prefix hyper_ suggests that they are top-level parameters that control the learning process and determine the values of model parameters that a learning algorithm ends up with.

Hyperparameters are values that ML engineer set their configuration even before the training process begins. Hyperparameters are said to be external to the model because the model cannot change its values during learning.

You can think of them like the knobs and dials on an old radio. You tune the different dials and buttons and hope that a nice tune comes out.

The algorithm does not learn the value of these during the modeling process. This is the crucial differentiator between hyperparameters and parameters. Whether you set it or the algorithm learns it and informs you.

Where To Find the Model Parameters?

To know what parameters an algorithm will produce, you need to Know a bit about the algorithm itself and how it works. And consult the Scikit Learn documentation to see where the parameter is stored in the returned object.

The parameters are found in the documentation for that particular algorithm under the Attributes Section, not the parameters section

3.1 Parameters in Logistic Regression

To make this concrete, consider a simple logistic regression model. We create the estimator and fit to the data with default settings. Since the logistic regression model is a linear model, we will get beta coefficients on our variables. These are found in the coef_ property of our logistic regression object. However, if we print these out, we can see it is a bit messy.

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split


x_train, x_test, y_train, y_test = train_test_split(x, y, test_size= 0.2)

logistic_regression_model = LogisticRegression()
logistic_regression_model.fit(x_train, y_train)
print(logistic_regression_model.coef_)

[[-3.32930871e-05 -3.75822277e-06 -6.58898668e-07 -7.90989416e-07
  -7.37626915e-07 -1.01104425e-05  1.82449947e-06  1.42557118e-06
   1.24483955e-06  1.18426156e-06  1.10402558e-06  1.02111887e-06
  -1.13331931e-05  4.54146176e-06  2.18016350e-06  2.83747495e-06
   5.26357246e-06  1.38630721e-06 -3.83575714e-05 -1.92088805e-05
  -1.13513317e-05 -1.38235492e-05 -3.12894603e-06 -1.39180198e-06]]

Let us clean this up by creating a list of original variable names, zipping this up with the coefficients and formatting into a neat DataFrame for easy viewing.

original_variables = list(x_train.columns)
zipped_together = list(zip(original_variables, logistic_regression_model.coef_[0]))
coefs = [list(x) for x in zipped_together]
coefs = pd.DataFrame(coefs, columns=["Variable", "Coefficient"])

coefs_sorted = coefs.sort_values(by=["Coefficient"], axis=0, ascending=False)
coefs_sorted.head(10)

We can now sort the DataFrame and print the top 10 results for brevity. Do you recall setting PAY_0 to have a coefficient of 0.000002 ? I don’t.

Variable Coefficient
BILL_AMT5 0.000005
BILL_AMT2 0.000005
BILL_AMT4 0.000003
BILL_AMT3 0.000002
PAY_0 0.000002
PAY_2 0.000001
BILL_AMT6 0.000001
PAY_3 0.000001
PAY_4 0.000001
PAY_5 0.000001

The coefficients are parameters because we did not set them ourselves and were learned during the modeling process. In our data, the PAY variables relate to how many months people have previously delayed their payments. We can see that having a high number of months of delayed payments, makes someone more likely to default next month.

Parameters in tree-based models.

Tree-based models e.g., decision tree and Random forest, don’t have linear coefficients. The parameters of this model are in the nodes of the trees used to build the model such as what feature was split on and at what value.

Get AI & Data Science 604’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

To demonstrate, let us firstly build a random forest estimator & fit to our data, setting the max_depth to be quite low only for visualization purposes. Then we can pull out a single tree, found in the random forest estimator ‘estimators_’ attribute to visualize. For simplicity we will just show the image but you can explore visualizing this yourself using the mentioned packages.

from sklearn.ensemble import RandomForestClassifier

random_forest_model = RandomForestClassifier(max_depth= 3)
random_forest_model.fit(x_train, y_train)
chosen_tree = random_forest_model.estimators_[7]

Here, we see a graph of the nodes including the variables and values used in the splits. We can see that the very first split was on the variable PAY_4 and it sent samples left or right depending if they had a value above or below 1 for this variable. Do you remember setting this decision? I certainly don’t!

Where to Find the Hyperparameters?

We can easily see the hyperparameters by creating an instance of the estimator and printing it out.

Random_Forest_classifier = RandomForestClassifier()
print(Random_Forest_classifier)

Here we create the estimator with default settings and call the print function on our estimator.

Those are all our different knobs and dials we can set for our model. There are a lot! But what do they all mean?
For this we need to turn to the Scikit Learn documentation. I.e., the n_estimatorshyperparameter. We can see in the documentation that it tells us the data type, and the default value and it also provides a definition of what it means.

Why do we need to tune the hyperparameters?!

As always in the ML world, there’s no universal “one-size-fits-all” approach. Default hyperparameters are generic settings that may not work optimally for every specific problem or dataset.
Therefore, it’s crucial to systematically adjust hyperparameters to find the configuration that maximizes a model’s performance for a particular task. This process, known as hyperparameter tuning, helps ensure that the machine learning model is tailored to the unique characteristics of the problem at hand.

The Hyperparameter Importance

Some are more important than others. But before we outline important ones, there are some hyperparameters that definitely will not help model performance. These are related to computational decisions or what information to retain for analysis. I.e., With the random forest classifier, n_jobs=None How many cores to use will only speed up modeling time? verbose= 0 whether to print out information as the modeling occurs also won’t assist. These hyperparameters will not assist model performance.

Hence some hyperparameters you don’t need to ‘train’ during your work.

There are some generally accepted important hyperparameters to tune for a Random Forest model. I.e., the n_estimators(how many trees in the forest) should be set to a high value, 500 or 1000 or even more is not uncommon (noting that there are computational costs to higher values) The max_featurshyperparameter, controls how many features to consider when splitting, which is vital to ensure tree diversity.
max_depth and min_sapmples_split featurscontrol overfitting of individual trees. Thecriterion hyperparameter may have a small impact but it is not generally a primary hyperparameter to consider.

Remember, this is just a guide and your particular problem may require attention on other hyperparameters.

How to Find hyperparameters that matter?

There are hundreds of ML algorithms out there and learning which hyperparameters matter is knowledge you will build over time from a variety of sources. I.e., there are some great academic papers where people have tried many combinations of hyperparameters for a specific algorithm on many datasets. These can be a very informative read! You can also find great blogs and tutorials online and consult the Scikit Learn documentation. Of course, one of the best ways to learn is just more practical experience! It is important you research this yourself to build your knowledge base for efficient modeling.

Some hyperparameters are likely better to start your tuning with than others. What we didn’t discuss was what values should you try. This will be specific to the algorithm and to the hyperparameter itself
But there does exist best practice around this. Let’s walk through some top tips for deciding ranges of values to try for different hyperparameters.

Conflicting Hyperparameter Choices

It is firstly important to know what values NOT to set as they may conflict. You will see in the Scikit Learn documentation for the Logistic Regression Algorithm, that some values of the hyperparameter ‘penalty’ conflict with some values of the hyperparameter ‘solver’.
Another example from the ElasticNet algorithm demonstrates a softer conflict that will not result in an error but may result in a model construction we had not anticipated. Safe to say, close inspection of the Scikit Learn documentation is important.

There’re also values for different hyperparameters that may be valid but are very unlikely to yield good results.
Some examples of this are: Having a random forest algorithm with a very low number of trees. Would you consider it a forest if it had 2 trees? How about 5 or 10? Still probably not. But at 300, 500, 1000 or more that is definitely getting there!

Having only 1 neighbor in a K-nearest neighbor algorithm. This algorithm averages votes of ‘neighbors’ to your sample. Safe to say averaging the vote of 1 person doesn’t sound robust!

Finally, incrementing some hyperparameters by a small amount is unlikely to greatly improve the model. I.e., One more tree in a forest, isn’t likely to have a large impact. Researching and documenting sensible values for different hyperparameters and algorithms will be a very useful activity.

Automating Hyperparameter Choice.

you built several different models to test a single hyperparameter like so. This was quite an inefficient way of writing code, I think we can do better to test different values for the number of neighbors hyperparameter.

knn_5 =  KNeighborsClassifier(n_neighbors=5)
knn_10 = KNeighborsClassifier(n_neighbors=10)
knn_20  = KNeighborsClassifier(n_neighbors=20)

This is quite inefficient. Can we do better? One thing we could try is using a for loop (Manul Search).

from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score


neighbors_list = [3,5,10,20,50,75]
accuracy_list = []
for test_number in neighbors_list:
    model = KNeighborsClassifier(n_neighbors=test_number)
    predictions = model.fit(x_train, y_train).predict(x_test)
    accuracy = accuracy_score(y_test, predictions)
    accuracy_list.append(accuracy)

accuracy_list

First, we create a list of values to test. Then loop through the list, creating an estimator, fitting and predicting each time. Lastly, we append the accuracy to a list of accuracy scores to analyze after. This method easily allows us to test more values than our previous work.

[0.7246443322625057,
 0.7400183570445158,
 0.768242312987609,
 0.7691601652134007,
 0.7726021110601193,
 0.7739788893988068]

We can store the results in a Data Frame to view the effect of this hyperparameter on the accuracy of the model. It appears that adding any more neighbors don’t help beyond 20.

Learning Curves

A common tool that is used to assist with analyzing the impact of a singular hyperparameter on an end result is called a ‘learning curve’.
Firstly, let’s create a list of many more values to test using Python’s range function. The rest of the code is the same as before.

from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score

neighbors_list = list(range(5,500, 5))
accuracy_list = []
for test_number in neighbors_list:   
   model = KNeighborsClassifier(n_neighbors=test_number)    
   predictions = model.fit(x_train, y_train).predict(x_test)    
   accuracy = accuracy_score(y_test, predictions)    
   accuracy_list.append(accuracy)
results_df = pd.DataFrame({'neighbors':neighbors_list, 'accuracy':accuracy_list})

Since we tested so many values, we will use a graph rather than a table to analyze the results. We plot the accuracy score on the Y axis and our hyperparameter value on our X axis.

import matplotlib.pyplot as plt
plt.plot(results_df['neighbors'],     results_df['accuracy'])
plt.gca().set(xlabel='n_neighbors', ylabel='Accuracy', 
title='Accuracy for different n_neighbors')
plt.show()

We can see our suspicions confirmed, that accuracy does not increase at all beyond where we tested before.

One thing to be aware of is that python’s range function does not work for decimal steps which is important for hyperparameters that work on that scale. A handy trick uses NumPy’s linspace function that will create a number of values, evenly spread between a start and end value that you specify. Here is a quick example you can see. 5 values, evenly spaced between 1 and 2 inclusive.