Neural Network HyperParameters

What are Hyperparameters?

Hyperparameters are variables that determine the network structure and how the network is trained. They include changes in the number of layers,change in number of neurons,the type of activation function used in each layer,weights initialization logic,the optimizer to use, the learning rate,the batch size, and are tuned to get better model performance while training.

• Learning Rate: It controls the step size taken by the optimizer during each training iteration. Too small learning rate can slow convergence and too large learning rate can lead to instability and divergence.
• Epochs: It represents number of time the entire training dataset passes through the model during training. Increasing the number of epochs can improve model performance but sometimes may lead overfitting as well
• Number of Layers: It determines depth of model,which can have significant impact on the model complexity and learning ability
• Number of Nodes per layer: It determines width of model, influencing the number of layer and connection between layers. The optimal structure depends on complexity of task and size of dataset.
• Activation Function: It introduces non-linearity to the model,allowing to learn complex decision boundaries.

import keras_tuner as kt
import keras


def build_model(hp):
  n_hidden= hp.Int("n_hidden",min_value=0,max_value=8,default=2)
  n_neuron= hp.Int("n_neurons",min_value=16,max_value=256)
  learning_rate= hp.Float("learning_rate",min_value=1e-4,max_value=1e-2)
  optimizer= hp.Choice("optimizer",values=['sgd','adam'])
  if optimizer == 'sgd':
    optimizer= keras.optimizer.SGD(learning_rate=learning_rate)
  else:
    optimizer= keras.optimizer.Adam(learning_rate=learning_rate)
  model = keras.Sequential()
  model.add(keras.layer.flatten())
  for _ in range(n_hidden):
    model.add(keras.layers.Dense(n_neuron,activation='relu'))
  model.add(keras.Dense(10,activation='relu'))
  model.compile(loss='sparse_categorical_crossentropy',optimizer=optimizer,metrics=['accuracy'])
  return model

random_search_tuner= kt.RandomSearch(build_model,objective="val_accuracy",max_trials=5,overwrite=True,   #it builds model for 5 trials usinh hyperparameters with respective range and train for 10 epochs
                                     directory= "my_fashion_minst",project_name='my_rnd_search',seed=42)
random_search_tuner.search(X_train,y_train,epochs=10,validation_data=(X_vaid,y_valid))

top3_params= random_search_tuner.get_best_models(num_models=3)
best_model= top3_params[0] #{'n_hidden':5,'n_neuron':70,'learning_rate':0.0052454,'optimizer':'adam'}

best_model.fit(X_train,y_train,epochs=10)

Number of Hidden Layer

• A model with fewer hidden layers will extract low level structures(line segments or shapes,orientations in image recognition)
• Intermediate hidden layer will combine lower level and intermediate level structures(i.e squares,circles in image recognition)
• Highest level layers and output layer will combine these intermediate structures to high level structures(i.e faces in image recognition)

Number of Neuron Per Layer

• Number of neurons in input and output layer is determined by the type of input and output your task requires.(e.g., the MNIST task requires 28x28= 784 inputs and 10 output neurons.
• Start with more neurons in the first layer, allowing low-level features to coalesce into fewer higher-level features(i.e first with 300 neurons,second with 200 and third with 100)
• Depending on dataset, it can sometimes help to make the first hidden layer bigger than others

Learning Rate

• Optimal learning rate is half of maximum learning rate
• One way to find good learning rate is to train the model for a few hundred iterations,with very low learning rate and gradually increasing it to a larger value. This is done by multiplying learning rate by constant factor at each iteration(e.g 10/10^-5)¹/500) to go from 10^-5 to 10 in 500 iterations.
• Optimal learning rate will be a bit lower than the point at which loss starts to climb.Then can reintialize the model and train it normally using good learning rate

BatchSize

• It can have significant impact on model’s performance and training time
• The main benifit of using larger batch size is that hardware accelerators like GPUs process them efficiently, so training algorithm will see more instances per second
• Larger batch sizes often lead to instabilities,especially at the begining of training. So it can be generalize that model is trained with small batch size

Epoch

• Epoch represents one complete pass through entire training dataset.
• Imagine you have book of information for training your model. An epoch signifies going through entire book once.
• The number of times you read the book determines how many time model sees the whole dataset
• Training for more epochs allows the model to learn more complex data

Training Step

• A training step refers to single update of model’s internal parameters based on batch of data
• Think of chapter in training book. A training steps signifies processing small chunk of information from the chapter
• Each epoch consists of many training steps, depending on the batch size and size of training set

Analogy between Training Step,Batch Size and Epoch

Imagine learning a new language

An epoch would be like going through entire language book at once

A training step would be like studying single page or section from textbook

The number of times you go though the textbook(epoch) determines how well you will learn the language

How you study each section(batch size) can impact how efficiently you learn the material

How Batch Size is Calculated?

If the batch size is set to 32 (which is often the default), it means the model processes 32 training samples at a time.

Get Mansi Mehta’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

For example, if your training dataset has 55,000 images, the number of batches per epoch will be:

55,000 ÷ 32 ≈ 1,719 batches

During training:

• The model picks 32 images from the dataset.
• It performs a forward pass, calculates the loss, and then updates its internal parameters using backpropagation.
• This process repeats for each batch until the entire dataset is processed — that is, 1,719 times per epoch in this case.

Better activation functions

• Mostly, the ReLU activation function is the most suitable. activation function because it does not saturate for positive values
• But relu activation is not perfect all the time because it has problem of dying relu: during training some neurons effectively die,meaning stop outputting anything other than 0, especially if you use large learning rate.
• A neuron dies when its weights get tweaked in such way that input of relu function is negative for all instance of training set.
• When this happens it just keeps outputing zero and gradient descent doesn’t affect it anymore because the gradient of relu function is zero when its output is negative
• This problem is solved by Leaky relu

Batch Normalization

• It is used in reducing vanishing/exploding gradient problems
• The technique consists of adding an operation to the model just before and after activation function of hidden layer
• The operation simply zero centers and normalize each input, then scales and shifts the result using two new parameter vectors per layer: one for scaling and one for shifting
• Operation lets model learn the optimal scale and mean of each layer’s input
• If you add batch normalization as the first layer, then there is no need to standardize your training set. There is no need of StandardScaler or Normalization
• In order to zero-center and normalize inputs, the algorithm needs to estimate each input’s mean and standard deviation.
• It does so by evaluating mean and standard deviation of input over the current batch
• BN standardize input, then rescales and offsets them

#implementation of batch normalization in keras
import tensorflow as tf

model= keras.Sequential([
    keras.layers.Flatten(input_shape=[28,28]),
    keras.layers.BatchNormalization(),
    keras.layers.Dense(300,activation='relu',kernel_initializer='he_normal'),
    keras.layers.BatchNormalization(),
    keras.layers.Dense(100,activation='relu',kernel_initializer='he_normal'),
    keras.layers.Dense(10,activation='softmax')
])
model.summary()

Gradient Clipping

• It is used in exploding gradient problem is to clip the gradient during backpropagation so that they never exceed some threshold. This is Gradient Clipping
• This is generally used in RNN, where use of batch normalization is tricky.
• In keras, implementing gradient clipping is just matter of setting the clipvalue ot clipnorm argument when creating optimizer

optimizer= tf.keras.optimizers.SGD(clipvalue=1.0)
model.compile([...],optimizer= optimizer)

Regularization to avoid Overfitting

• Deep neural networks typically have tens of thousands of parameters.
• This gives them incredible amount of freedom and means they can fit huge variety of complex datasets.
• But these great fit also makes netwrok prone to overfitting training set.
• Regularization is often needed to prevent overfitting.
• Regularization terms add a penalty term to the model’s loss function.
• The goal of training neural network is to minimize loss function. By adding peanalty, we are not only trying to minimize the error on training data but also trying to keep model’s complexity in check.
• This complexity is often related to the magnitude of the connection weights.

When a neural network overfits, it often develops very large weights. These large weights mean that small changes in input can lead to large changes in output, making the model highly sensitive to the training data’s specific features and noise, rather than learning general patterns. Regularization helps to “shrink” these weights.

1. L1 Regularization(Lasso Regularization): This technique adds a penalty to loss function based on the magnitude of model’s weights. The goal is to discourage overly complex models by forcing weights to be smaller.
2. L2 Regularization(Ridge Regression/Weight Decay): It add penalty term proportional to the sum of squares of weights (sumw_i^2) to the original loss function
3. Dropout: Dropout is a highly effective and widely used regularization technique specific to neural networks. During training, for each training batch, a random subset of neurons (along with their connections) in a layer are temporarily “dropped out” or ignored. This means their outputs are set to zero for that specific forward and backward pass. The probability of dropping a neuron is given by the dropout rate (p).

Learning Rate Scheduling

• Finding good learning rate is very important.
• If you set it too high, training diverges and if you set it too low then training will eventually converge to optimum,but it will take very long time.
• If it is slightly too high, it will progress very quickly at first but it will end up dancing around the optimum and never settle down.
• If you find good learning rate by training the model for a few hundred iterations, exponentially increasing learning rate from smaller value to larger value, then looking at the curve and picking a learning rate slightly lower than the one at which the learning curve starts shooting back up
• If you can start with larger learning rate and then reduce it once training stops making progress faster,you can reach good solution faster than with the optimal constant learning rate
• There are many strategies for reducing the learning rate during training. This is called learning rate scheduling

1. Power Scheduling
2. Exponential Scheduling
3. Piecewise constant scheduling
4. Performance Scheduling
5. 1cycle Scheduling

Key Considerations for Choosing Batch Size

Memory Constraints:

• Small Batch Sizes (e.g., 16, 32) require less memory and are suitable for machines with limited resources.
• Large Batch Sizes (e.g., 256, 512) need more memory but can accelerate training if you have access to high-end GPUs or TPUs.

Training Stability:

• Small Batch Sizes lead to noisy gradients which can help in escaping local minima but might cause instability during training.
• Large Batch Sizes result in more stable gradients and faster convergence, but can lead to convergence at sharp minima which might not generalize well.

Training Speed:

• Small Batch Sizes might be slower because more updates are required to complete an epoch.
• Large Batch Sizes can speed up the training process by reducing the number of updates required per epochs

How to select Batch Size?

• Start with a Moderate Batch Size: Begin with a size like 32 or 64. This is generally a good starting point and provides a balance between stability and training speed.
• Increase Gradually: If you have the computational resources, gradually increase the batch size and observe if it improves performance.
• Use Batch Size as a Hyperparameter: Treat batch size as a hyperparameter to be tuned along with learning rates and other parameters.
• A larger batch size reduce number of steps per epochs
• A small batch size increase number of steps per epochs

Key Considerations for Choosing Number of Epochs

Avoid Overfitting:

• Too Few Epochs: Can lead to underfitting where the model doesn’t learn enough from the data.
• Too Many Epochs: Can lead to overfitting, where the model starts to memorize the training data rather than generalizing from it.

Early Stopping:

• Monitoring Validation Performance: Implement early stopping to halt training when the model’s performance on a validation set stops improving.

Learning Rate and Batch Size:

• Higher Learning Rates: Often require fewer epochs as larger steps are taken in parameter space.
• Smaller Learning Rates: Generally require more epochs because smaller steps lead to a more gradual convergence.

How to select Number of Epochs?

• Start with a Base Value: Begin with 50 or 100 epochs as a baseline and adjust based on performance.
• Use Early Stopping: Track validation loss or accuracy and stop training when there’s no improvement for a set number of epochs.
• Experiment with Epochs: Try different values and use cross-validation to find the optimal number of epochs for your specific model and dataset.

Finding the Balance Between Batch Size and Epochs

• Smaller Batch Sizes might require more epochs to achieve the same level of performance as larger batch sizes due to noisier gradient estimates.
• Larger Batch Sizes can speed up training and potentially reduce the number of epochs required but might lead to overfitting if not monitored properly.