Automatic Hyperparameter Optimization and Feature Selection with HYBparsimony Package

How to do it in just a few lines of Python code

Performing feature selection (FS) and hyperparameter optimization (HO) at the same time is much more efficient than doing it separately. HYBparsimony not only allows you to do this in a very simple way, but it also takes care of finding the simplest model (parsimonious model).

Search for Parsimonious Models in Small and High-Dimensional Data Sets with HYBparsimony Package

Want an accurate model with as few variables as possible? Use the HYBparsimony Python package.

HYBparsimony is a Python package that simultaneously performs automatic: feature selection (FS), model hyperparameter optimization (HO), and parsimonious model selection (PMS).

First, install the package.

pip install hybparsimony

Get Francisco Javier Martínez de Pisón Ascacíbar’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

HYBparsimony Python package has been published under the MIT license. The repository with more examples and information can be found at: https://github.com/jodivaso/hybparsimony/.

Then, run this code and go to sleep.

import pandas as pd
from sklearn.model_selection import train_test_split, RepeatedKFold
from sklearn.metrics import mean_squared_error
from sklearn.datasets import load_diabetes
from sklearn.preprocessing import StandardScaler
from hybparsimony import HYBparsimony

# Load 'diabetes' dataset
diabetes = load_diabetes()
X, y = diabetes.data, diabetes.target
X_train, X_test, y_train, y_test = train_test_split(X, y, 
    test_size = 0.2, random_state=1234)

# Standarize X and y
scaler_X = StandardScaler()
X_train = scaler_X.fit_transform(X_train)
X_test = scaler_X.transform(X_test)
scaler_y = StandardScaler()
y_train = scaler_y.fit_transform(y_train.reshape(-1,1)).flatten()
y_test = scaler_y.transform(y_test.reshape(-1,1)).flatten()

# Note: The results of the examples may vary depending on the 
# hardware available.
algorithms_reg = ['Ridge', 'Lasso', 'KernelRidge', 'KNeighborsRegressor', 
'MLPRegressor', 'SVR', 'DecisionTreeRegressor', 'RandomForestRegressor']

res = []
for algo in algorithms_reg:
  print('#######################')
  print('Searching best: ', algo)
  # Search the best hyperparameters and features
  HYBparsimony_model = HYBparsimony(algorithm=algo,
       features=diabetes.feature_names,
       rerank_error=0.001,
       cv=RepeatedKFold(n_repeats=10, n_splits=5),
       n_jobs=10, # each job executes one fold
       maxiter=1000,
       verbose=1)
  
  # Set 'time_limit=0.20' (12secs) to check the code
  HYBparsimony_model.fit(X_train, y_train, time_limit=60) # 60 minutes!!!
  
  # Check results with test dataset
  preds = HYBparsimony_model.predict(X_test)
  print(algo, "RMSE test", mean_squared_error(y_test, preds, squared=False))
  print('Selected features:',HYBparsimony_model.selected_features)
  print(HYBparsimony_model.best_model)
  print('#######################')

  # Append results
  res.append(dict(algo=algo,
      MSE_5CV= -round(HYBparsimony_model.best_score,6),
      RMSE=round(mean_squared_error(y_test, preds, squared=False),6),
      NFS=HYBparsimony_model.best_complexity//1e9,
      selected_features = HYBparsimony_model.selected_features,
      best_model=HYBparsimony_model.best_model))

res = pd.DataFrame(res).sort_values('RMSE')
# Visualize results
print(res[['best_model', 'MSE_5CV', 'RMSE', 'NFS', 'selected_features']])

When you wake up, you will get a table similar to the one below showing the best models found for each algorithm (results may vary depending on the hardware available). In this case, the model that best generalized the problem was an MLP with only 6 input features out of 10 and a single neuron in the hidden layer!!!

Best parsimonious models for ‘diabetes’ dataset.

How the Code Works

See Searching Parsimonious Models in Small and High-Dimensional Datasets with HYBparsimony Python Package and more examples in package’s GitHub repository: https://github.com/jodivaso/hybparsimony.

Reference

Divasón, J., Pernia-Espinoza, A., Martinez-de-Pison, F.J. (2023). HYB-PARSIMONY: A hybrid approach combining Particle Swarm Optimization and Genetic Algorithms to find parsimonious models in high-dimensional datasets. Neurocomputing, 560, 126840. 2023, Elsevier. https://doi.org/10.1016/j.neucom.2023.126840