Activation Function

An activation function is a mathematical function used in artificial neural networks to decide whether a neuron should be activated (turned on) or deactivated (turned off) based on its input.

Think of it as a switch that determines if a neuron should pass the signal forward in the network or not.

Why Do We Need an Activation Function?

Without an activation function, a neural network would behave like a linear equation, meaning it could only solve simple problems.
📌 Activation functions make neural networks capable of learning complex patterns by introducing non-linearity.

Types of Activation Functions

1️⃣ Linear Activation Function

✅ Formula: f(x) = ax

• The output is directly proportional to the input.
• Problem: It cannot handle complex data (only works for simple tasks).
• Not used in deep learning.

2️⃣ Non-Linear Activation Functions

Most activation functions used in deep learning are non-linear. These functions allow networks to learn complex patterns in data.

(a) Sigmoid Function (Logistic Function)

✅ Formula:

Sigmoid Function (Logistic Function)

📌 How it works:

• Squeezes output between 0 and 1.
• Good for binary classification (Yes/No, True/False).

❌ Problems:

• Vanishing gradient — Small changes in input make almost no difference in output.
• Not good for deep networks.

(b) Tanh (Hyperbolic Tangent) Function

✅ Formula:

Tanh (Hyperbolic Tangent) Function

📌 How it works:

• Squeezes output between -1 and 1.
• Works better than Sigmoid for deep networks.

❌ Problem: Still suffers from vanishing gradient.

(c ) ReLU (Rectified Linear Unit) Function

✅ Formula:

ReLU (Rectified Linear Unit) Function

📌 How it works:

• Outputs zero for negative values.
• Passes positive values as they are.
• Most popular activation function in deep learning.

❌ Problem: Dying ReLU — Some neurons stop working if they get negative inputs too often.

Get Vikas Jindal’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

✔ Solution: Leaky ReLU, which allows small negative values.

(d) Softmax Function (For Multi-Class Classification)

✅ Formula:

Softmax Function

📌 How it works:

• Converts multiple outputs into probabilities.
• Used in the last layer of a multi-class classification model.

Which Activation Function to Use?

Activation Function Use Case Sigmoid Binary classification (e.g., spam vs. not spam) Tanh When outputs need to be between -1 and 1 ReLU Most deep learning models (CNNs, RNNs, etc.) Leaky ReLU Solves the dying ReLU problem Softmax Multi-class classification

Example: Activation Function in Python

import numpy as np
import matplotlib.pyplot as plt

# Define activation functions
def sigmoid(x):
    return 1 / (1 + np.exp(-x))
def relu(x):
    return np.maximum(0, x)
# Generate input values
x = np.linspace(-10, 10, 100)
# Compute outputs
y_sigmoid = sigmoid(x)
y_relu = relu(x)
# Plot the functions
plt.plot(x, y_sigmoid, label="Sigmoid")
plt.plot(x, y_relu, label="ReLU")
plt.legend()
plt.show()

Summary

✔ Activation functions introduce non-linearity in neural networks.
✔ ReLU is the most popular activation function in deep learning.
✔ Softmax is used for multi-class classification.
✔ Sigmoid and Tanh suffer from the vanishing gradient problem.