Why we initialize weight?

Weight initialization helps a lot in optimization for deep learning. Without it, SGD and its variants would be much slower and tricky to converge to the optimal weights.
The aim of weight initialization is to prevent layer activation outputs from exploding or vanishing during the course of a forward pass through a deep neural network.

Now Let’s understand different ways to initialize the weight.

1. Zero Initialization

Here we are initializing all weights as 0. In this case, all neurons will compute the same thing.

a.Here we are sending the same output to every neuron in the network.
It will output the same thing. So we can say the neuron is not learning.

b.Even we initialize constant weight. Then same gradient update will be there.
These above issues lead to the problem of “Symmetry”.

2.Initializing weight as large negative numbers.
Here we are initializing all weights as large negative numbers.

This is the graph of ReLU. So for the negative number output is 0.

2. Case 2 → Activation function as sigmoid.

In this case, for very large negative or positive weights. The gradient is very small. Then we have the problem of Vanishing Gradient.

Ideal conditions for weights
a.Weights should be not too small
b. Weights should not be 0.
c.Weights has some good variance. With this, each neuron will learn different things.

Now taking the above points into consideration, we will try to initialize the weight.

Case 1:Normal Initialization :
All the weight from Normal Distribution with mean = 0 & variance = small value.

Advantage :
a.Weight has the same variance.
b. We have some positive & negative weights.

Some better weights initialization strategy.

Before proceeding, let's understand two terms “fan_in” & “fan_out”

2. Uniform Initialization
It means we can initialize the weights that follow “Uniform Distribution” with
the minimum value & maximum values as follows-

3. Xavier / Glorot Initialization
Xavier's initialization has two variations. We can now check those variations.

a. Variation 1: All weights will come from Normal distribution with mean = 0 and variance for every neuron will be-

This initialization is known as Glorot Normal or Xavier Normal.

b.Variation 2: Xavier/Glorot Uniform.
All weights will come from the uniform distribution with the below conditions.

Xavier initialization works well with Sigmoid activation.

4. “He” Initialization
It also has two variations. Lets find out

a. Variation 1: He- Normal
All weights will come from Normal distribution with mean = 0 and variance for every neuron will be-

b.Variation 2: He- Uniform
All weights will come from the uniform distribution with the below conditions.

“He” Initialization works fairly well with ReLU & Leaky ReLu.

Conclusion:
If you guys have any doubts please feel free to comment down below. I hope you guys enjoyed this little interesting journey of weight initialization.

Connect with me- https://www.linkedin.com/in/paritosh-kumar-3605a913b/

References:
https://www.appliedaicourse.com/

Weights Initialization in Neural Network was originally published in Analytics Vidhya on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you ever face a scenario like this, Congrats it means you have huge data :D :D.
Knowing the Computational complexity is very important in Machine Learning.

So my topic is, “What are the computational complexities of ML Models”

Time complexity can be seen as the measure of how fast or slow an algorithm will perform for the input size. Time complexity is always given with respect to some input size (say n).
Space complexity can be seen as the amount of extra memory you require to execute your algorithm. Like the time complexity, it is also given with respect to some input size (n).

The complexity of an algorithm/model is often expressed using the Big O Notation, which defines an upper bound of an algorithm, it bounds a function only from above.
The graph below visualizes different cases of complexities for algorithms.

To write computation complexity we are assuming as,
n= number of training examples, d= number of dimensions of the data,
k= number of neighbors

The complexity of K Nearest Neighbors to find the k closest neighbor

Train Time Complexity = O(knd)
Loops through every training observation and computes the distance d between the training set observation and new observation.

Time is linear with respect to the number of instances (n) and dimensions (d).

Space Complexity = O(nd)
K Nearest Neighbors store the data.
Testing takes longer because you have to compare every test instance to the whole training data.

The complexity of Logistic Regression

Training Time Complexity means in logistic regression, it means solving the optimization problem.
Train Time Complexity=O(nd)

Space Complexity = O(d)
Note: Logistic regression is very good for low latency applications.

The complexity of SVM

Training Time Complexity=O(n²)
Note: if n is large, avoid using SVM.

Run-time Complexity= O(k*d)
K= number of Support Vectors,d=dimentionality of the data

The complexity of Decision Tree

Training Time Complexity= O(n*log(n)*d)
n= number of points in the Training set
d=dimentionality of the data

Run-time Complexity= O(maximum depth of the tree)
Note: We use Decision Tree when we have large data with low dimensionality.

The complexity of Random Forest

Training Time Complexity= O(n*log(n)*d*k)
k=number of Decision Trees
Notes: When we have a large number of data with reasonable features. Then we can use multi-core to parallelize our model to train different Decision Trees.

Run-time Complexity= O(depth of tree* k)
Space Complexity= O(depth of tree *k)
Note: Random Forest is comparatively faster than other algorithms.

The complexity of Naive Bayes

Training Time Complexity = O(n*d)
Run-time Complexity = O(c*d)
We have to retrieve feature for each classes ‘c’

Conclusion:
If you have a large amount of data , choosing the algorithm is based on the business problem that you are going to solve. Try to reduce the dimension of the data if required to decrease the computation complexities.
If you guys have any doubts please feel free to comment down below.
Connect with me- https://www.linkedin.com/in/paritosh-kumar-3605a913b/

References :
https://www.appliedaicourse.com/
https://stats.stackexchange.com/questions/96995/machine-learning-classifiers-big-o-or-complexity
https://imgflip.com/

Time Complexity of ML Models was originally published in Analytics Vidhya on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article, I will present a technique that help us to predict the anomaly in the data.

Anomaly is an unexpected change within these data patterns, or an event that does not conform to the expected data pattern, is considered an anomaly. In other words, an anomaly is a deviation from business as usual.

Condition monitoring (CM) is the process of monitoring a parameter of conditions in machinery (vibration, temperature, etc.), in order to identify a significant change which is indicative of a developing fault.

Technical Aspects :
After understanding the data from the business, We come across a new algorithm. So anomalous behavior for this pattern may tell about the failure.

So the main question comes,” How we can find the future anomaly data”

After doing Exploratory Data Analysis, we come across that 10 parameters are impacting the output variable.

So for finding the future anomaly data point, I have written one algorithm.

Algorithm for finding the future Anomaly :

1. In this approach, we will convert our problem into a “Regression Problem”. Then the “output” parameter will become the dependent variable and the other 10 parameters as the independent variable.
2. Sort the data set based on timestamps. Convert the timestamps into pandas datetime format and sort.
3. Take the time lag for each independent variable and calculate the correlation with the dependent variable respectively. Check the code snippet.

3. Use the above-mentioned code for all the important parameters to find the correlation between the respective “Output” parameter and the lagged independent variable.

For Example, Let name the independent variable as “Y” and one dependent variable as X. Let suppose we found that (X-10)the value is highly correlated to current variable “Y”, it means that “Y” is highly correlated with past X-10th value which means for value of “Y” at 10.00 pm is highly correlated with value of “X” at 9.55 pm(10*30 seconds[time gap between data]).

4. Likewise we can find the lagged based correlation for other parameters also.

5. Develop a “Regression” based machine learning model with lagged data as an independent variable and “Output” parameter as dependent and deploy the model.

6. With this approach, we will pass the previous data points to find the “Output” value for the future unseen variable.

7. Once we have the future data points, we can tell that the data is an anomaly or not using Z-score, IQR etc.

Conclusion:
If you guys have any doubts please feel free to comment down below. I have not gone into too much code for the approach as I don’t want to increase the length of the article I hope you guys enjoyed this little interesting journey of the anomaly detection.

Connect with me- https://www.linkedin.com/in/paritosh-kumar-3605a913b/

References :
https://www.appliedaicourse.com/
https://towardsdatascience.com/how-to-use-machine-learning-for-anomaly-detection-and-condition-monitoring-6742f82900d7

Anomaly Detection using lagged based approach algorithm was originally published in Analytics Vidhya on Medium, where people are continuing the conversation by highlighting and responding to this story.