I was doing some research about sparsity in neuronal network framework and I came around the article “Group sparse regularization for deep neural networks”. This paper addressed the challenging task of simultaneously optimizing (i) the weights of a neural network, (ii) the number of neurons for each hidden layer, and (iii) the subset of active input features (i.e., feature selection). They essentially proposed an integration of lasso penalties in the regularization framework in the neuronal network. There is plenty of posts and extensive research on the definition and the importance of regularization. For example these posts [1] and [2] are a quick introduction to the regularization subject.

In order to reproduce the experiments in “Group sparse regularization for deep neural networks”, I decided to build my project using PyTorch which is a well known deep learning framework. According to Medium, this is the 4th best deep learning framework. It is gaining more and more popularity in academic levels since it’s easily modulable to test new ideas faster than most of its competitors.

I realize there is not such tools available to do any kind of regularization unless the L2-regularization. In pyTorch, the L2 is implemented in the “weight decay” option of the optimizer unlike Lasagne (another deep learning framework), that makes available the L1 and L2 regularization in their built-in implementation. You can find a lot of proposed solutions explaining how to implement the L1 regularization using the pyTorch framework but there is not easy implementation tool available to plug in your experimentation.

To compensate this absence, I decided to build some “ready to use” regularization object using the pyTorch framework. The implementation can be found here. I implemented the L1 regularization , the classical L2 regularization, the ElasticNet regularization (L1 + L2), the GroupLasso regularization and a more restrictive penalty the SparseGroupLasso, introduced in Group sparse regularization for deep neural networks.

The class object is built to have the pyTorch model as a parameter. We can apply the regularization penalty on specific parameter weights or on all the weights of every layer in the model.

I applied these implementations in the case of a fully connected neuronal network. This is the most simple suited model architecture to see the impact of the different types of regularization on the model performances.

Like the original article, I tested the implementation on the Digits dataset which represents an image classification problem. It is composed of 1797 8 × 8 grey images of handwritten digits collected from several dozens different people. I used a simple fully connected neural network with two hidden layers having respectively 40 and 20 neurons.

Looking at the results we can see that the L2-regularized model and the Weight Decay regularized model have the same performances on the test set. This is pretty good and what we expected since the Weight Decay is the L2 implementation already proposed by pyTorch. The sparser methods (L1-regularized and GL-regularized models) perfom quite well too but they are not better than the Weight Decay regularized model. We can now look at the real sparsity of the model. To study this we fixed that weights under 10e-3 in absolute value are set to 0. And then we analyzed all the % of sparsity of the neurons in each model. The % of sparsity is calculated by taking the number of neuron that have all their weights summed up at 0 and divide that number by the size of each layer. As expected, the L1-regularized model have sparser density than the L2-regularized model. But the performance still look great in this case. Sometimes, too much sparsity is not good for the model too.

In conclusion I made available a built-in and adaptable implementation of different type of regularization options. I tested all the regularization on a simpler fully connected model on the Digits datasets in order to see the performance between the different regularizer. We can see that sparsity in those case still perform pretty well on the task. The next step is to see how we can extend the pyTorchReg module to other model architecture.

Different types of regularization On Neuronal Network with Pytorch was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.