Pruning in Machine Learning
Model Optimisation
What is Pruning ?
Weight pruning is an optimization method that reduces the number of parameters by removing low-weight connections used to increase sparsity.
Basic Concept of Sparsity
Sparsity refers to the proportion of zeros in data or matrix. It can be thought of as the ratio of weights that are zero in a network of model. A common metric used to quantify sparsity is the L0 norm, which represents the number of non-zero elements in a vector.
For example, in a 3x3 matrix with 5 non-zero elements and 4 zero elements, the sparsity is calculated as the number of zeros divided by the total number of elements
Compressed Sparse Row (CSR)
Sparse matrices is inefficient since most of values in matrix are zero. CSR is sparse matrix storage format and it reduces memory usage by storing only the non-zero values and their positions. Using the CSR method, we can only access the necessary elements to compute.
Values : Values which are used to computations.
Get Youngjin(Jay) Na’s stories in your inbox
Join Medium for free to get updates from this writer.
Column indices: Identification which columns require multiplication.
Row pointers: Identification of the start & end points of each row.
- Row pointers[i] indicate the index where the i-th row starts in the Values array. The last value represents the length of the Values array.
- Row pointers[0] the values are stored from Values[0] to Values[3–1].
- Row pointers[1], the values are stored from Values[3] to Values[4–1].
- Row pointers[2], the values are stored from Values[4] to Values[5–1].
Excessive use of pruning can risk lowering the model’s accuracy. Therefore, it is essential to find a balance between model weight Pruning and accuracy.
Reference
Jinsol Kim, Sparse matrix & CSR(Compressed Sparse Row)
