Introduction

In the rapidly evolving field of data science, there is a growing interest in leveraging advanced techniques like Large Language Models (LLMs) for AI applications. While LLMs have garnered attention, it’s essential to remember that tabular data remains a cornerstone, requiring advanced techniques to maximize model performance. This article explores the integration of LightGBM, MLflow, and Optuna in a unified workflow, focusing on optimizing model parameters without delving into feature engineering. The analysis centers on the renowned Titanic dataset, a tabular dataset with the binary survival outcome.

Titanic Dataset Overview

The Titanic dataset is a classic dataset widely used for machine-learning exercises. For the purpose of this study, we will focus on specific features: Pclass, Sex, Age, SibSp, Parch, Fare, Embarked, and Survived. The goal is to build a model that predicts whether a passenger survived or not based on these features. It's important to note that we will not perform any feature engineering in this analysis, but rather concentrate on optimizing the LightGBM model through parameter tuning.

The dataset can be accessed and downloaded from Kaggle

Example Rows of Titanic Dataset:

| Pclass | Sex    | Age  | SibSp | Parch | Fare     | Embarked | Survived |
|--------|--------|------|-------|-------|----------|----------|----------|
| 3      | male   | 22.0 | 1     | 0     | 7.25     | S        | 0        |
| 1      | female | 38.0 | 1     | 0     | 71.2833  | C        | 1        |
| 3      | female | 26.0 | 0     | 0     | 7.925    | S        | 1        |
| 1      | female | 35.0 | 1     | 0     | 53.1     | S        | 1        |
| 3      | male   | 35.0 | 0     | 0     | 8.05     | S        | 0        |

Why LightGBM?

Before delving into model development, let’s understand why LightGBM is a powerful choice for this task. LightGBM, a gradient-boosting framework, stands out for its speed and efficiency. Its leaf-wise tree growth strategy and histogram-based approach make it well-suited for large datasets. The ability to handle categorical features and robust performance in diverse scenarios make LightGBM an essential tool in a data scientist’s arsenal.

Key Parameters and Capabilities of LightGBM:

• boosting_type: Specify the boosting method (e.g., gbdt, dart).
• num_leaves: Control the maximum number of leaves in one tree.
• learning_rate: Adjust the step size in each iteration.
• max_depth: Set the maximum depth of the tree.
• objective: Define the optimization objective (e.g., binary classification).

Initial Model Evaluation

Now, let’s proceed with the initial model evaluation. Before diving into hyperparameter optimization, we evaluate the performance of a basic LightGBM model on the Titanic dataset using the F1 score, a metric that balances precision and recall.

# Import necessary libraries
import pandas as pd
from sklearn.metrics import f1_score
from sklearn.model_selection import train_test_split
from sklearn.metrics import f1_score, confusion_matrix
import lightgbm as lgb
import optuna
import mlflow
import mlflow.lightgbm


# Load Titanic dataset
titanic_data = pd.read_csv('titanic.csv')  # Assuming the dataset is stored in 'titanic.csv'

# Select specific features
selected_features = ['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked', 'Survived']
titanic_data = titanic_data[selected_features]

# Convert categorical features to numerical using one-hot encoding
titanic_data = pd.get_dummies(titanic_data, columns=['Sex', 'Embarked'], drop_first=True)


# Extract features and target variable
X = titanic_data.drop('Survived', axis=1)
y = titanic_data['Survived']

# Split dataset
X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.2, random_state=42)

# Train an initial LightGBM model
initial_model = lgb.train({}, lgb.Dataset(X_train, label=y_train), 100)

# Make predictions and calculate F1 score
y_pred_initial = initial_model.predict(X_valid)
f1_initial = f1_score(y_valid, (y_pred_initial > 0.5).astype(int))

# Display F1 Score and Confusion Matrix
print(f"Initial F1 Score: {f1_initial}")
print("Confusion Matrix:")
print(confusion_matrix(y_valid, (y_pred_initial > 0.5).astype(int)))

After training the initial LightGBM model on the Titanic dataset, the model achieved an F1 Score of approximately 0.79. Breaking down the results further with a confusion matrix:

[[92 13]
[17 57]]

In the confusion matrix:

• True Positive (TP): 92
• False Positive (FP): 13
• False Negative (FN): 17
• True Negative (TN): 57

The Significance of Tabular Data and Parameter Tuning

In the contemporary AI landscape, Large Language Models (LLMs) often steal the spotlight, but the importance of tabular data and advanced techniques in maximizing model performance should not be overlooked. In this case study, we refrain from feature engineering and concentrate on optimizing the LightGBM model through parameter tuning.

MLflow for Experiment Tracking

MLflow simplifies the machine learning lifecycle by offering tools to manage experiments, track parameters, and store models. Its logging capabilities facilitate collaboration among team members, enabling reproducibility and efficient model comparison.

Key Capabilities of MLflow:

• Tracking Experiments: Log parameters, metrics, and artifacts to record and compare experiments.
• Model Packaging: Easily package and share models in a standardized format.
• Model Deployment: Simplify the deployment process with MLflow’s deployment options.

Optuna for Hyperparameter Optimization

Optuna automates the hyperparameter optimization process, helping data scientists find the best combination of parameters efficiently. Its framework supports various optimization algorithms, making it adaptable to different optimization scenarios.

Key Capabilities of Optuna:

• Distributed Optimization: Parallelize hyperparameter search across multiple nodes.
• Pruning: Accelerate optimization by early-stopping unpromising trials.
• Scalability: Scale up optimization efforts for complex models and large datasets.

Integrating LightGBM, MLflow, and Optuna

Below is a simplified example of integrating LightGBM with MLflow and Optuna using the Titanic dataset:

# Define LightGBM objective function for Optuna
def objective(trial):
    params = {
        'objective': 'binary',
        'metric': 'binary_error',
        'boosting_type': 'gbdt',
        'num_leaves': trial.suggest_int('num_leaves', 2, 256),
        'max_depth': trial.suggest_int('max_depth', 2, 32),
        'learning_rate': trial.suggest_loguniform('learning_rate', 0.005, 0.5),
        'feature_fraction': trial.suggest_uniform('feature_fraction', 0.1, 1.0),
    }

    # Set Train and Valid datasets
    lgb_train = lgb.Dataset(X_train, y_train)
    lgb_valid = lgb.Dataset(X_valid, y_valid, reference=lgb_train)

    # Train the model
    model = lgb.train(params, lgb_train, num_boost_round=100, valid_sets=[lgb_train, lgb_valid], 
                      callbacks=[lgb.record_evaluation({"valid": lgb_valid})])

    # Make predictions and calculate F1 score
    y_pred = model.predict(X_valid, num_iteration=model.best_iteration)
    f1 = f1_score(y_valid, (y_pred > 0.5).astype(int))

    # Log parameters and metrics with MLflow
    with mlflow.start_run():
        mlflow.log_params(params)
        mlflow.log_metric('f1', f1)
        # Save the model as an artifact
        mlflow.lightgbm.log_model(model, "model")

    return f1
# Optimize hyperparameters with Optuna
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)

1. Objective Function Definition:

• The objective function defines the hyperparameters that Optuna will optimize.
• It sets up a dictionary (params) containing hyperparameters such as the objective, metric, boosting type, number of leaves, max depth, learning rate, and feature fraction.

2. Dataset Preparation:

• LightGBM datasets (lgb_train and lgb_valid) are created using the training (X_train, y_train) and validation (X_valid, y_valid) data.

3. Model Training:

• The LightGBM model is trained with the current set of hyperparameters using the training dataset (lgb_train).
• The model is validated on the validation dataset (lgb_valid), and training progress is recorded with the record_evaluation callback.

4. Prediction and Metric Calculation:

• Predictions are made on the validation set, and the F1 score is calculated using the f1_score function.

5. Logging with MLflow:

• A new MLflow run is started, and the hyperparameters (params) and F1 score are logged.
• The trained LightGBM model is saved as an artifact with the name “model.”

6. Optimization with Optuna:

• The Optuna study is created to optimize the objective function (maximize F1 score) over a specified number of trials (n_trials).

This process iteratively refines hyperparameters to find the combination that maximizes the F1 score. The final model and optimal hyperparameters are then logged using MLflow.

Tracking with MLflow

To view model runs and metrics logged in MLflow, follow these steps:

1. Install MLflow: pip install mlflow
2. Start the MLflow UI: Run mlflow ui in your terminal.
3. Open your browser and go to http://localhost:5000 to access the MLflow UI.
4. Explore the recorded runs, metrics, and parameters.

Logging models with MLflow provides an organized record of each iteration’s configuration, facilitating easy comparison and decision-making. The MLflow UI acts as a user-friendly control center, offering insights into model performance and aiding in the seamless exploration and utilization of trained models.

Training the Final Model

Training once more with the optimal parameters identified by the Optuna process, we establish our ‘final_model.’ Now equipped with the best parameters, this final model is primed for evaluation on the test data.

# Train the final model with the best hyperparameters
best_params = study.best_params
final_model = lgb.train(best_params, lgb.Dataset(X_train, label=y_train), num_boost_round=100)

# Make predictions and calculate F1 score
y_pred_final = final_model.predict(X_valid)
f1_final = f1_score(y_valid, (y_pred_final > 0.5).astype(int))

# Display F1 Score and Confusion Matrix
print(f"Final Model F1 Score: {f1_final}")
print("Confusion Matrix:")
print(confusion_matrix(y_valid, (y_pred_final > 0.5).astype(int)))

After optimizing the LightGBM model using the Optuna hyperparameter tuning technique, the final F1 Score achieved is approximately 0.81

Confusion Matrix:

[[92 13]
[15 59]]

In the confusion matrix:

• True Positive (TP): 92
• False Positive (FP): 13
• False Negative (FN): 15
• True Negative (TN): 59

The F1 Score of 0.81 signifies a noteworthy enhancement compared to the initial model’s performance. Notably, through the streamlined Optuna, MLflow, and LightGBM process, we achieved a significant accuracy boost without delving into any intricate feature engineering efforts. This underscores the effectiveness of our approach in maximizing model accuracy through strategic parameter tuning.

Conclusion

In conclusion, this case study illustrates the significance of advanced techniques in maximizing model performance with tabular data. By leveraging LightGBM, MLflow, and Optuna, the article demonstrates a streamlined workflow for optimizing model parameters without engaging in feature engineering. The focus on parameter tuning in this scenario provides insights into the potential improvements achievable through more intricate feature engineering. As the AI industry embraces new buzzwords like Large Language Models (LLMs), it remains imperative to tackle tabular data with advanced techniques for robust model development.

Data manipulation and analysis are crucial tasks in many fields, it is important to be able to quickly and easily manipulate and analyze large datasets. Two popular tools for data manipulation and analysis are Python pandas and dplyr R.

While these two tools have some similarities, they are designed and used differently. In this article, we will compare pandas and dplyr, highlighting their key differences and showing how they can be used to solve common data manipulation problems using common and advanced capabilities such as filter, arrange, group by, select, join, and window functions.

We will use made-up housing data to show the capabilities of each tool.

Filter

One common data manipulation task is to filter data based on certain criteria. Both pandas and dplyr provide tools for filtering data.

Here is an example of how to use pandas to filter housing data:

import pandas as pd

# load the data from a CSV file
df = pd.read_csv('housing_data.csv')

# filter the data frame to only include houses with a price greater than $500,000, at least 3 bedrooms, and a square footage greater than 1,500
df = df[(df['price'] > 500000) & (df['bedrooms'] >= 3) & (df['sq_ft'] > 1500)]

# view the first few rows of the filtered data
df.head()
#   house_id     date     price  bedrooms  sq_ft  bathrooms  zipcode
#        3   1/3/2020  1020000         3   1600          2    90210
#        4   1/4/2020  1030000         3   1700          2    90210
#        5   1/5/2020  1040000         3   1800          2    90210
#        6   1/6/2020  1050000         3   1900          2    90210

Here is an example of how to use dplyr to filter housing data:

# install and load the dplyr package
install.packages("dplyr")
library(dplyr)

# load the data from a CSV file
df <- read.csv('housing_data.csv')

# filter the data frame to only include houses with a price greater than $500,000, at least 3 bedrooms, and a square footage greater than 1,500
df <- df %>%
  filter(price > 500000, bedrooms >= 3, sq_ft > 1500)

# view the first few rows of the filtered data
head(df)
#   house_id     date     price  bedrooms  sq_ft  bathrooms  zipcode
#        3   1/3/2020  1020000         3   1600          2    90210
#        4   1/4/2020  1030000         3   1700          2    90210
#        5   1/5/2020  1040000         3   1800          2    90210
#        6   1/6/2020  1050000         3   1900          2    90210

Arrange

Another common data manipulation task is to sort data based on certain criteria. Both pandas and dplyr provide tools for sorting data.

Here is an example of how to use pandas to sort housing data:

# sort the data frame by the 'price' column in descending order, and then by the 'bedrooms' column in ascending order
df = df.sort_values(['price', 'bedrooms'], ascending=[False, True])

# view the first few rows of the sorted data frame
df.head()
#   house_id     date     price  bedrooms  sq_ft  bathrooms  zipcode
#        6   1/6/2020  1050000         3   1900          2    90210
#        5   1/5/2020  1040000         3   1800          2    90210
#        4   1/4/2020  1030000         3   1700          2    90210
#        3   1/3/2020  1020000         3   1600          2    90210

Here is an example of how to use dplyr to sort housing data:

# sort the data frame by the 'price' column in descending order, and then by the 'bedrooms' column in ascending order
df <- df %>%
  arrange(desc(price), bedrooms)

# view the first few rows of the sorted data frame
head(df)
#   house_id     date     price  bedrooms  sq_ft  bathrooms  zipcode
#        6   1/6/2020  1050000         3   1900          2    90210
#        5   1/5/2020  1040000         3   1800          2    90210
#        4   1/4/2020  1030000         3   1700          2    90210
#        3   1/3/2020  1020000         3   1600          2    90210

Group by

Another common data manipulation task is to group data based on certain criteria and then perform operations on the groups. Both pandas and dplyr provide tools for grouping data.

Here is an example of how to use pandas to group housing data:

# group the data frame by the 'zipcode' column and calculate the mean price for each group
df = df.groupby('zipcode')['price'].mean()

# view the resulting data frame
df
# zipcode
# 90210    1040000
# 90211    1050000
# 90212    1020000

Here is an example of how to use dplyr to group housing data:

# group the data frame by the 'zipcode' column and calculate the mean price for each group
df <- df %>%
  group_by(zipcode) %>%
  summarise(mean_price = mean(price))

# view the resulting data frame
df
# zipcode  mean_price
# 90210    1040000
# 90211    1050000
# 90212    1020000

Select

Another common data manipulation task is to select specific columns or rows from a data frame. Both pandas and dplyr provide tools for selecting data.

Here is an example of how to use pandas to select columns from a housing data frame:

# select the 'date', 'price', and 'zipcode' columns from the data frame
df = df[['date', 'price', 'zipcode']]

# view the first few rows of the resulting data frame
df.head()
#      date   price    zipcode
#   1/3/2020  1020000    90210
#   1/4/2020  1030000    90210
#   1/5/2020  1040000    90210
#   1/6/2020  1050000    90210

Here is an example of how to use dplyr to select columns from a housing data frame:

# select the 'date', 'price', and 'zipcode' columns from the data frame
df <- df %>%
  select(date, price, zipcode)

# view the first few rows of the resulting data frame
head(df)
#    date    price    zipcode
#  1/3/2020  1020000    90210
#  1/4/2020  1030000    90210
#  1/5/2020  1040000    90210
#  1/6/2020  1050000    90210

Join

Another common data manipulation task is to join data from multiple sources. Both pandas and dplyr provide tools for joining data.

Here is an example of how to use pandas to join two housing data frames on the house_id column:

# load the second data frame from a CSV file
df2 = pd.read_csv('housing_data_2.csv')

# join the two data frames on the 'house_id' column, using an inner join to only include rows with matching values in both data frames
df = df.join(df2, on='house_id', how='inner')

# view the first few rows of the resulting data frame
df.head()
#   house_id     date     price  zipcode       city     state  year_built
#        3   1/3/2020  1020000    90210  Beverly Hills    CA        1950
#        4   1/4/2020  1030000    90210  Beverly Hills    CA        1960
#        5   1/5/2020  1040000    90210  Beverly Hills    CA        1970
#        6   1/6/2020  1050000    90210  Beverly Hills    CA        1980

Here is an example of how to use dplyr to join two housing data frames on the house_id column:

# load the second data frame from a CSV file
df2 <- read.csv('housing_data_2.csv')

# join the two data frames on the 'house_id' column, using an inner join to only include rows with matching values in both data frames
df <- df %>%
  inner_join(df2, by='house_id')

# view the first few rows of the resulting data frame
head(df)
#   house_id     date     price  zipcode       city     state  year_built
#        3   1/3/2020  1020000    90210  Beverly Hills    CA        1950
#        4   1/4/2020  1030000    90210  Beverly Hills    CA        1960
#        5   1/5/2020  1040000    90210  Beverly Hills    CA        1970
#        6   1/6/2020  1050000    90210  Beverly Hills    CA        1980

There are more joining operations we can process using dplyr and pandas such as left_join,right_join, and full_join.

Window functions

Another advanced data manipulation capability is the use of window functions, which allow you to perform calculations across a set of rows related to the current row. Both pandas and dplyr provide tools for using window functions.

Here is an example of how to use pandas to calculate the difference between the current rows price and the previous rows price using a window function:

# define the window function that calculates the difference between the current rows 'price' and the previous rows 'price'
def price_diff(df):
  return df['price'] - df['price'].shift(1)

# apply the window function to the data frame, grouping by the 'zipcode' column
df = df.groupby('zipcode').apply(price_diff)

# view the first few rows of the resulting data frame
df.head()
#   house_id     date  price  zipcode       city        state  year_built price_diff
#        3  1/3/2020  1020000    90210  Beverly Hills    CA        1950       NaN
#        4  1/4/2020  1030000    90210  Beverly Hills    CA        1960     10000
#        5  1/5/2020  1040000    90210  Beverly Hills    CA        1970     10000
#        6  1/6/2020  1050000    90210  Beverly Hills    CA        1980     10000

Here is an example of how to use pandas to calculate the difference between the current rows price and the previous row's price using a window function:

# define the window function that calculates the difference between the current rows 'price' and the previous rows 'price'
price_diff <- function(df) {
  df %>%
    mutate(price_diff = price - lag(price))
}

# apply the window function to the data frame, grouping by the 'zipcode' column
df <- df %>%
  group_by(zipcode) %>%
  price_diff()

# view the first few rows of the resulting data frame
head(df)
#   house_id     date     price  zipcode       city     state  year_built price_diff
#        3   1/3/2020  1020000    90210  Beverly Hills    CA        1950        NA
#        4   1/4/2020  1030000    90210  Beverly Hills    CA        1960     10000
#        5   1/5/2020  1040000    90210  Beverly Hills    CA        1970     10000
#        6   1/6/2020  1050000    90210  Beverly Hills    CA        1980     10000

Other than using lag in window functions, there are more common functions to use like row_number, rank, and dense_rank. The usage of lag is simply to show how we can approach window function with pandas and dplyr.

Conclusion

In conclusion, pandas and dplyr are both powerful tools for data manipulation and analysis. While they have some similarities, they are designed and used in different ways, and each has its strengths and weaknesses. By understanding the key differences between pandas and dplyr, you can choose the right tool for your data manipulation and analysis tasks.

R dplyr vs Python pandas: A Showdown of Data Manipulation Powerhouses was originally published in Dev Genius on Medium, where people are continuing the conversation by highlighting and responding to this story.

XGBoost is a powerful and popular machine learning algorithm that uses an ensemble of decision trees to improve the predictive performance of a model. In this article, we’ll see how to use XGBoost to train and tune a model using the Titanic dataset, which contains information about the passengers of the Titanic, such as their age, sex, and fare class, and is often used to train machine learning models to predict whether a passenger survived the sinking of the ship or not.

Preparing the data

First, we need to import the necessary Python libraries and load the Titanic data into a DataFrame. We'll use the pandas library to load and manipulate the data.

import pandas as pd
from sklearn.model_selection import train_test_split

# Load the Titanic data into a Pandas DataFrame
df = pd.read_csv('titanic.csv')

# Print the first few rows of the DataFrame
print(df.head())

    PassengerId  Survived  Pclass ...     Fare Cabin  Embarked
0            1         0       3  ...   7.2500   NaN         S
1            2         1       1  ...  71.2833   C85         C
2            3         1       3  ...   7.9250   NaN         S
3            4         1       1  ...  53.1000  C123         S
4            5         0       3  ...   8.0500   NaN         S

Next, we need to select the features we want to use to train the model and the target variable. In this case, we’ll use the Age, Sex, Pclass ,Fare and Embarkedcolumns as features and the Survived column as the target variable. Then, we'll split the data into training and test sets using the train_test_split() function.

1. Handling missing values: The Titanic dataset may have missing values in some of the columns. These missing values can be handled by imputing them with a suitable replacement value, such as the column mean or median.
2. Encoding categorical variables: The dataset may contain columns with categorical values, such as the gender of a passenger (male or female). These values need to be encoded as numerical values in order for the model to be able to handle them. This can be done using techniques such as one-hot encoding.
3. Normalizing numerical values: The numerical values in the dataset may have different scales, such as the age of a passenger (measured in years) and the fare paid for the ticket (measured in dollars). This can cause some algorithms to weigh certain columns more heavily than others, which can affect the model’s performance. Normalizing the values to have a common scale can address this issue.

# One-hot encode Pclass feature
pclass_dummies = pd.get_dummies(df["Pclass"], prefix="Pclass")
# Drop original Pclass column and add one-hot encoded columns
df = df.drop("Pclass", axis=1)
df = pd.concat([df, pclass_dummies], axis=1)

# One-hot encode Embarked feature
embarked_dummies = pd.get_dummies(df["Embarked"], prefix="Embarked")
# Drop original Embarked column and add one-hot encoded columns
df = df.drop("Embarked", axis=1)
df = pd.concat([df, embarked_dummies], axis=1)

# handle missing values
df['Age'] = df['Age'].fillna(df['Age'].median())

# encode categorical variables
df['Sex'] = df['Sex'].map({'male': 0, 'female': 1})

# normalize numerical values
df['Age'] = (df['Age'] - df['Age'].mean()) / df['Age'].std()
df['Fare'] = (df['Fare'] - df['Fare'].mean()) / df['Fare'].std()


#Select the features we want to use to train the model
X = df[['Age', 'Sex', 'Fare','Pclass_1', 'Pclass_2', 'Pclass_3',
       'Embarked_C', 'Embarked_Q', 'Embarked_S']]

# Print the unique values in the sex column
print(X['Sex'].unique())

# Select the target variable
y = df['Survived']

# Split the data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Here you can find the titanic dataset https://www.kaggle.com/competitions/titanic/data

Training an XGBoost model

Now that we have prepared the data, we can train an XGBoost model using the xgboost library. First, we need to import the library and create an XGBClassifier object. Then, we can train the model using the fit() method, passing it the training data and the target variable. For the first training process, we will use the default parameter values.

# Import the XGBoost library
import xgboost as xgb

# Create an XGBoost model
model = xgb.XGBClassifier()

# Train the model on the training data
model.fit(X_train, y_train)

Once the model is trained, we can use it to make predictions on the test data and evaluate its performance using an appropriate metric, such as accuracy or F1 score. For this, we can use the accuracy_score() and f1_score() functions from the sklearn library to evaluate the model's performance.

from sklearn.metrics import accuracy_score, f1_score

# Use the model to make predictions on the test data
y_pred = model.predict(X_test)

# Evaluate the model's performance using accuracy
acc = accuracy_score(y_test, y_pred)
print('Accuracy:', acc)
# Accuracy: 0.7988826815642458

# Evaluate the model's performance using F1 score
f1 = f1_score(y_test, y_pred)
print('F1 score:', f1)
# F1 score: 0.7567567567567567

The accuracy_score() calculates the proportion of correct predictions made by the model and returns it as a decimal value between 0 and 1. The f1_score() function calculates the weighted average of the precision and recall of the model and returns it as a decimal value between 0 and 1. Higher values for both metrics indicate better-performing models.

Why F1 Score?

Precision is the fraction of true positives (i.e., the number of passengers who were correctly predicted to have survived) out of all positive predictions (i.e., the total number of passengers predicted to have survived). High precision means that the model is not predicting too many passengers to have survived when they actually did not, which is important because false positives (i.e., passengers who were predicted to have survived but actually did not) can lead to incorrect conclusions.

Recall is the fraction of true positives (i.e., the number of passengers who were correctly predicted to have survived) out of all actual positive cases (i.e., the total number of passengers who actually survived). High recall means that the model is correctly predicting most of the passengers who actually survived, which is important because false negatives (i.e., passengers who were predicted to not have survived but actually did) can lead to missed opportunities.

The F1 score is the harmonic mean of the precision and recall, which means that it considers both of these metrics equally.

Tuning hyperparameters

Tuning hyperparameters is the process of finding the optimal values of these parameters that result in the best performance of the model on a given dataset.

For example, suppose you want to tune the following parameters for an XGBoost classifier:

• max_depth: This parameter sets the maximum depth of the tree. Increasing the value of max_depth can lead to more complex and accurate models, but can also increase the risk of overfitting.
• learning_rate: This parameter sets the learning rate for the gradient boosting algorithm. Increasing the learning rate can lead to faster convergence, but can also lead to unstable models.
• n_estimators: This parameter sets the number of trees in the model. Increasing the value of n_estimators can lead to more accurate models, but can also increase the computational cost.
• gamma: This parameter sets the minimum loss reduction required to make a further partition on a leaf node of the tree. Increasing the value of gamma can lead to more conservative models, which can reduce the risk of overfitting.
• subsample: This parameter sets the fraction of observations to be used for each tree. Increasing the value of subsample can lead to more accurate models, but can also increase the computational cost.

GridSearchCV performs a grid search over a specified range of hyperparameter values and finds the combination that results in the best performance.

from xgboost import XGBClassifier
from sklearn.model_selection import GridSearchCV

# Create an XGBoost model
model = XGBClassifier()

# Define the grid of hyperparameters we want to search
hyperparameters = {
    'max_depth': [3,4,5,6,7],
    'learning_rate': [0.1, 0.3, 0.5],
    'n_estimators': [100, 200,300],
    'gamma': [0,0.1,0.3, 1],
    'subsample': [0.5,0.7, 1.0]
}

# Use GridSearchCV to tune the hyperparameters of the model
model = GridSearchCV(model, hyperparameters, cv=3, scoring='f1')
model.fit(X_train, y_train)

Here you can learn all xgboost parameters: https://xgboost.readthedocs.io/en/stable/parameter.html

Once the grid search is completed, we can access the best-performing model using the best_estimator_ attribute and use it to make predictions on the test data.

# Print the best combination of parameters
print(model.best_params_)

{'gamma': 1, 'learning_rate': 0.1, 'max_depth': 5, 'n_estimators': 200, 'subsample': 0.5}

# Use the best-performing model to make predictions on the test data
y_pred = model.best_estimator_.predict(X_test)

# Evaluate the model's performance using F1 score
f1 = f1_score(y_test, y_pred)
print('F1 score:', f1)
# F1 score: 0.7832167832167832

Using a hyperparameter technique improved our model F1 score compared to a default parameter model without any tunning process.

Conclusion

In this article, we saw how to use XGBoost to train and tune a model using the Titanic dataset. We prepared the data, trained a basic XGBoost model, and used GridSearchCV to tune the hyperparameters of the model. The resulting model achieved an F1 score of 0.783 on the test data, which indicates that it was able to make accurate predictions about whether a passenger survived the sinking of the Titanic or not.

There are many ways in which the performance of the model can be improved further. For example, we could try using more features from the dataset, such as the number of siblings and parents they had on board, or we could try using different evaluation metrics, such as precision or recall. Additionally, we could try using different ensemble methods, such as random forests or gradient-boosting machines, to improve the predictive power of the model.

If you want to apply the techniques shown in this article to your own datasets and problems, you can use the code provided as a starting point and experiment with different hyperparameters and features to see how they affect the performance of the model. With some practice and tuning, you’ll be able to use XGBoost to improve the accuracy of your predictions and solve your own machine-learning problems.