Hyperparameters are special kind of parameters that are trained generally before or during training to determine model’s complexity, rate of learning and how model should learn and hence should be determined and found before the model has been trained
Press enter or click to view image in full size
Hyperparameter tuning is the process of finding the best hyperparameter for this very purpose
Get Akshat Gupta’s stories in your inbox
Join Medium for free to get updates from this writer.
Various methods like :
RandomSearch : To search randomly by selecting random samples of hyperparameter combination to avoid doing the computationally expensive search through all parameters combination like GridSearch, this can be useful especially in numerous ML models as ususally majority of hyperparameters are do not contribute heavily and have very less effect on Models
GridSearch : Searches through all combination of hyparameter settings to include all kinds of combination and select the best model available from that
Manual Search : Manually selecting value of Hyperparameter and then training model on that and then evaluating by repeating this process multiple times for different values of Hyperparameter selected manually
Bayesian Search : Important search algorithm invovling learning from previous model’s output and using a surrogate model that sequentially initially trains some models using some intial hyperparameter values and then making a surrogate model and a system which evaluates the results of these previous model’s to optimise and avoid repeating them, then the model selects best possible next set of hyperparameter values to repeat the same process, and with a better and effective if found updates the model
Also we should use Hyperparameter tuning with validation phase of model training and with validation set, and first train model with training set, then use validation set and hyperparameter tuning to update and compare models, usually using k-fold cross validation which invovles dividing the dataset into k -folds and then using k-1 folds as training set and kth fold as validation set and then repeating the process k times each time selecting different validation set, also we should be careful that there is no data leakage and no test set is ever leaked into this process otherwise it can lead to overfitting easily, finally we should use test set to evaluate perfomance and accuracy of model and generalization of model on unseen data
