Sitemap
Alejando Attento

Alejando Attento

Data Scientist, Gaming and Finance https://www.linkedin.com/in/alejandro-daniel-attento/

XGBoost using JAVA explained

9 min readFeb 6, 2023

--

Most Data Scientists build their models in Python and R. Only in very specific cases it is necessary to use another language (a more efficient one). For example, if you want to make very fast inferences (Predictions) on a streaming of data.

Here we will see a simple example of how we could set up the pipeline for a Machine Learning model implemented with Java. From extracting the data from .zip files to evaluating the results. Also, you will see the differences between Java and Python when building a model.

In Python you have many libraries which provide many tools you can use to split your dataset, encode categorical labels or features, etc. That’s not the case with Java. Java requires you to build all of these functions by yourself. Luckily, here you will find a complete example of how to code a Machine Learning pipeline using JAVA.

What are we going to do?

We will take the Dry Bean Dataset (Link here) and build an XGboost model to predict to which class each bean belongs given a set of features. We will do this using Java and Maven.

As a quick introduction, Maven is a build tool that allows us to manage Java applications across different IDEs. Basically, it defines a specific way of how we should install and import all the libraries, how to structure our application, etc. If you use IntelliJ or Eclipse to develop your Java projects it’s already integrated.

Also, we are not going to spend much time doing any exploratory analysis, feature engineering, optimizing the model, etc. The purpose of this article is to show and demonstrate how to build s machine learning processes using Java.

Application overview

Here you can see the structure of our application. Notice the pom.xml file with all the instructions Maven needs to build our application. Don’t worry if you aren’t used to Maven, you don’t need to know it to understand the rest of the code. This article is not about working with Maven. The reason why I chose to use Maven is that if you’re used to work with Java most likely you know Maven. This will make it easier for you to copy the code and reproduce the results.

Besides the pom.xml file, let’s look at the structure. First you will see that in the src/main/java folder you have three different folders and the main class. Here is a quick description of these:

dataModels: This folder includes classes that will allow us to manage our data within the application. Basically, they define the format in which we are going to store and manage our data. Throughout the article, we will only explain one of them since the concept is exactly the same for the other two.

  1. RawDatasource: To store the training data we mentioned above.
  2. ProcessedData: To store the results after inferring the bean type.
  3. ResultData: To store the aggregated results data.

mlModels: Here are our Machine Learning model classes. For this example, there is only one.

  1. BasicXGBoostModel: Simple implementation of the XGBoost model.

utils: Here we have all the helper functions we will need for our application:

  1. DataChecks: This general class will allow us to implement different data checks. For example, it checks for null values. If we have rows with null values they are removed.
  2. Encoder: This class allows us to encode any categorical feature or label into numbers. Also, it generates a Json file to be able to decode these values later.
  3. JsonReader: This class allows us to read Json files and cast them as dictionaries to access the values.
  4. ModelEvaluation: This class runs tests on the model after it has been trained to evaluate its performance. For training and testing data, it assesses general performance as well as performance by bean type.
  5. Reader: This class allows us to read files as CSV files.
  6. TrainTestSplit: This class allows us to split the data we have between training and testing data.
  7. TransformToLibsvm: This class allows us to transform our data into Libsvm format. The XGBoost library requires data to be in that format.
  8. Unzip: This class allows us to extract data from .zip files.

Main.java: The main class that runs the whole application.

Get Alejando Attento’s stories in your inbox

Join Medium for free to get updates from this writer.

On the other hand you have the resources folder where we have all the data we need which are not Java classes or the pom.xml file. Here you will find:

  1. basicXGBoostModelConfig: A Json file with all the parameters required by the XGBoost model.
  2. dataset: We will keep all the data we are working on in this folder.
  3. DryBeanDataset: .zip file with the initial training data.

The Application

Before you look into the code of the application, here is a graph to summaries the steps.

Press enter or click to view image in full size

The whole application is managed by the main class, Main.java.

Main.java

We will step through the different parts of the Main class to understand what this application does step by step. Below you can find the code. As you might notice this application takes only two parameters:

  • The path to the zip file with the data to train the model.
  • A boolean to indicate if we want the application to print at which step of execution it is.

Unzip.java

The first thing this application does it to take the .zip file with the data and extract it into a CSV file. This is the only part of the code I did not write myself, so credit goes to the original author. You can see references in the code.

Reader.java

Once we have extracted all that data we need to read the CSV file. To do that we are going to use the reader function below.

But, once we did that the question is, in which format do we handle the data inside our application? To solve that we have the RawDatasource class.

Reader.java

We will read each row of the CSV file using an instance of the RawDatasource class. All these instances will be stored inside an arrayList. Find below the structure of these instances.

We show only this data model as an example, but the other two work in the same way. At the end we will show the complete code.

DataChecks.java

Once this is done our application is able to work with the data. The most common task is to perform checks on the data. Here I created a class that we could use to do many checks. In order to keep it short, I applied just a quick check to see if there are any null values. Those rows will be removed.

Encoder.java

Looking at the data you will see that all the features are numerical, but not the labels. Each row is labeled as one of 7 different types of beans like dermason, sira, etc. Basically, each bean has a name, but we need to encode these names into numbers. This is necessary because the XGBoost model does not take anything else than numbers as input, we can’t just pass strings to it. To do so we built an encoder function as below.

In addition to updating the data, these functions will also create a Json file to save how we encoded the data, this is to decode these values afterwards. If not, when we try to infer the type of bean based on the features we would get back a number which represents a class. However, we wouldn’t know what it was. Below you can see an example of the Json file:

TrainTestSplit.java

Once we have our data cleaned and encoded we need to split it into our testing and training datasets. For this, as for many of the things we did before, we use pre-built functions in Python. With Java, we have to create our own classes to manage it.

In this case we split our data 80% for training and 20% for testing.

TransformToLibsvm.java

Usually at this point we would train the model with our data. But, the xgboost4j library we use to train the XGBoost model requires the data to have a specific format called Libsvm.

If you are unfamiliar with that format, it’s quite simple to understand. It’s also known as the dense matrix format in some cases. The first column indicates the label of the data. Each of the following elements is preceded by the number of the column. After that, you have the actual value for that column. It becomes more clear in the example below.

To be able to come to this format we had to build our own class that handles it.

Using this we will create two libsvm files, one for the training data and another for the testing data.

BasicXGBoostModel.java

Now, with the proper format we can go forward and train the model. The parameters for the model are in a Json file as we said above. We are not attempting to optimize parameters here, that is a subject for another article. But, feel free to copy the code, set up the model and play around with the parameters to test out other parameters. These parameters are in the file called basicXGBoostModelConfig.

Given these parameters, we will pass it all to the model class, together with the data, to train the model.

This part can take more or less time depending on the parameters, the data, the iterations, etc. But, with the default settings, it should be fairly quick. Also, keep in mind that by default we set the epochs to just 50.

Finally, once the model is trained we save it as a bin file so that we can access it later. That’s basically what would happen in a real-world scenario.

You will find that model as model.bin once it runs.

ModelEvaluation.java

Finally, we have to evaluate our model. To do this, we built a class. We will evaluate the performance in general and for all of the individual bean classes. Of course, we are planning to do this for the testing and training datasets.

Once this part runs, we will receive the results of our model. Here you can see the results I got after the first run.

Notice that the Default probability is the probability of guessing the right kind of bean randomly given that you have 7 possible types. As you can see, the model performs quite well.

Here we should do many more analysis to understand better the model, to do a proper evaluation. But, the goal of this article is to show how to built a Machine Learning model using Java, not to do all the proper analysis and fine tuning a model requires.

Finally, you will see that the model returns us the bean class name rather than the associated value we got by encoding. This is because we saved in a Json file the encoding relations, this allows us to decode the data. The function calls that Json and translate the encoded value back to the name of the bean type.

Conclusion

Building a Machine Learning model with Java takes much more time than using Python or R. Also, we will have less resources than we are used to with Python, all the base libraries like pandas, scikit-learn, etc. to manage data and make data handling easier are not provided, we have to built our own tools. But, in some circumstances Java, because how fast it is can be the right option for our needs.

However, you will see that in some cases Python and Java are used together. An example could be training the model with Python, optimizing the parameters, etc. encode that model using a format that allows us to restore it in Java, and regenerate it in Java already trained. In that way we can take advantage of the speed Java has to do inferences, but train the model with Python

To access the complete code please refer to this link. Also, feel free to reuse this code for your own purposes but please do not forget to give credit.

If you have questions or problems running this model feel free to reach me out.

--

--