Activation Functions
- Activation Functions are mathematical gates that introduce non-linearity to a neural network, allowing it to learn complex, non liear relationships rather than acting as a giant , glorified linear regression model.
- It determines where a neuron should be activated or not and non linearity into neural network
- Neuron first calculated a weighted sum of its input
→ z = ∑ n to (i=1) (wi * xi +b)
→ or in vector form z = w^T *x+b
- The activation function is then applied to this value:
→ a = f(z)
where:
→ xi = input features
→ wi = weights
→ b = bias
→ z = weighted sum
→ f = activation function
→ a = neuron output
therefore, he general form of a neuron is:
a = f (∑ n to (i=1) (wi * xi +b))
- No activation function ==> Limited Representation capability
- Linear Transformation + Non Linearity = Powerful Neural Network
Major Role of Activation Function:
1. Introducing “Non Linearity” ( eg : Image Classification)
2. Controlling neuron output
3. Enable Deep Learning
Input — -> Low level features → High level features → Predictions
Major Types of Activation Functions
Binary Step activation function:
→ It is the one of the simplest activation function
→ It produces either 0 or 1
→ the mathematical definition is :
f(x) = { 1 if x>= 0, else 0}
f(x) belongs {0,1}
Advantages:
- Simple
- Easy to understand
- useful for binary classification
Disadvantages:
- The major problem is that the derivative is almost every where undefined at zero
- Binary step is generally not used in modern deep neural networks.
2. Linear Activation Function :
Linear activation function is: f(x) = x
f(x) = ax+b
Derivative : f′(x) = 1
Range :
−∞ < f(x) < +∞
Advantages:
- Simple
- No Saturation
- Useful for regression output layers
Disadvantage:
if you use only linear functions in your hidden layers, your “deep” network becomes a single, basic formula. No matter how many hidden layers you add, a network with only linear activation functions can never learn anything morecomplex than a straight line.
3. Sigmoid Activation Function:
- The sigmoid function is one of the most well-known activation functions.
σ(x) = 1/(1+e^-x)
Derivative : σ′(x) = σ(x)(1 −σ(x))
Range: 0 < σ(x) <1
Advantages :
- O/P is bounded between 0 and 1
- useful for probability based outputs
- Sigmaoid is commenly used for “Binary Classification”.
Disadvantages :
- Vanishing Gradient Problem
- Computationally More expensive than ReLu
- Saturation for very large positive or negative I/P
4. Hyperbolic Tangent (Tanh):
Tanh(x) = (e^(x) — e^(-x)) / (e^(x) + e^(-x))
Derivative : tanh′(x) = 1 −tanh2(x)
Range: -1 < tanh(x) <1
Advantages:
- Zero-centered output
- Stronger gradient than sigmoid around zero
Disadvantages :
- Vanishing Gradient Problem
5. ReLU– Rectified Linear Unit :
ReLu is the Most popular activation function in deep learning.
f(x) = max(0,x)
f(x) = { 0 if x<0 else x
Derivative :
Get Bhargav’s stories in your inbox
Join Medium for free to get updates from this writer.
f’(x) = { 0 if x<0 , 1 if x>0
Advantages :
- Faster than sigmoid and tanh
- Helps reduce the vanishing gradient problem for +ve I/p
- Computationally simple
- Produce Sparce activation
Disadvantages :
- Dying ReLu Problem : If a neuron consistently receives negative I/Ps, its output becomes zero and its gradient is also
zero. then neuron may stop learning.
6. Leaky ReLU :
Leaky ReLu helps to reduce the dying ReLu problem.
f(x) = { αx if x < 0 else x
α is small positive constant ( α = 0.01)
Advantages :
- Unlike ReLU, Leaky ReLU allows a small gradient when:
→ x <0
- Therefore neurons are less likely to become permanently inactive.
Disadvantages :
- we need tune α using hyperparameter
- α it is fixed value.
7. Parametric ReLU (PReLU) :
PReLu is similar to Leaky ReLu , but the negative slope is learne during training.
f(x) = { αx if x<0 else x
Advantages :
- In LReLu
→ Leaky ReLU: α is fixed
- PReLu
→ PReLU: α is learned
Disadvantages :
- We need to tune extra parameters.
8. Exponential Linear Unit (ELU) :
It is a non-linear activation function designed to make neural network training faster and more stable.
f(x) = { x if x>0 else α(e^x −1)
Advantages :
- Allows negative outputs
- Smooth for negative I/P
- can reduce vanishing gradient problem
Disadvantages:
- More Expensive than ReLU.
9. Scaled Exponential Linear Unit (SELU) :
It is a variant of the ELU activation function that introduces a highly unique capability: self-normalization.
f(x) = λ { x if x>0 else α (e^x — 1)
λ ≈1.0507, α ≈1.6733 These are constants.
10. Softplus :
Softplus is a smooth approximation of ReLU.
f(x) = ln(1 +ex)
SoftPlus behaves like ReLU:
→ Softplus(x) ≈ ReLU(x)
Derivative : f′(x) = 1 / (1 +e^(−x))
Advantages :
- Smooth
- Similar to ReLU
Disadvantages :
- More Expensive than ReLU.
11. Swish :
The Swish activation function is a smooth, non-monotonic activation function discovered by Google researchers in 2017 using automated search techniques. This can sometimes provide useful optimization behavior in deep neural networks.
f(x) = xσ(x)
σ(x) = 1 / (1 + e^(x))
f(x) = x / (1 + e^(x))
12. GELU– Gaussian Error Linear Unit :
The Gaussian Error Linear Unit (GELU) is a high-performance activation function introduced by Dan Hendrycks and Kevin Gimpel in 2016. It has become the industry standard for modern Natural Language Processing (NLP) and Transformer architectures.
GELU(x) = xΦ(x)
13. Softmax Activation Function :
The Softmax Activation Function is a unique function used almost exclusively in the very last layer (output layer) of a neural network designed for multi-class classification.
Suppose a neural network produces final ouputs like a1,a2,a3,… ak then Softmax function converts them into probabilities:
P(y=i) = e^zi / ∑ k to (j=1) e^zj
Then the outputs are:
0 < P(y=i) <1
and
∑ k to (i=1) P(y = i) = 1
Example:
Input : [ Dog, Cat, Horse]
Their outputs : [0.10,0.75,0.15]
Therefore:
0.10+0.75+0.15=1
Then the model assign the largest probability to the second class belongs to cat
