Activation Functions

Pothula Vineeth

20 min readOct 9, 2026

1. Introduction to Activation Functions

What is an activation function?

An activation function is used to introduce the non linearity to neural network and also controls the neural output.

Where is an activation function used in a neural network?

It is mainly used after the weighted sum of neurons.

Input layer → [Linear: z = Wx + b] → [Activation: a = f(z)] → [Linear] → [Activation] → … → Output

What happens if we do not use an activation function?

Without activation functions, stacking multiple neural-network layers does not give the network the ability to learn complex non-linear relationships.

Why can’t a neural network simply use the weighted sum as its output?

Because the weighted sum is a linear operation

y = w₁x₁ + w₂x₂ + b

This can only represent relatively simple linear relationships

But a neural network with non-linear activation functions can learn boundaries

What is non-linearity, and why is it essential for neural networks?

Non-linearity means the relationship between input and output is not simply a straight-line relationship

Real-world problems are usually not purely linear.

Examples:

Image recognition
Speech recognition
Object detection
Face recognition
NLP
Fraud detection
Medical diagnosis

The relationships between inputs and outputs can be extremely complicated.

Non-linear activation functions allow neural networks to model these relationships.

How do activation functions enable neural networks to learn complex patterns?

Input → Linear transformation→ Linear transformation → Linear→ transformation

2. Mathematical Foundation

1. How does a neuron calculate its output?

Step 1: weighted sum (pre-activation) z = w₁x₁ + w₂x₂ + … + wₙxₙ + b

Step 2: activation a = f(z) Term Meaning

What each term represents

A simple example

Take x₁ = 2, x₂ = 3, w₁ = 0.5, w₂ = 0.2, b = 0.1:

z = (0.5)(2) + (0.2)(3) + 0.1 = 1 + 0.6 + 0.1 = 1.7

With ReLU: a = max(0, 1.7) = 1.7
With Sigmoid: a = 1 / (1 + e^(−1.7)) ≈ 0.846

Same z, different outputs. The activation decides how the evidence is turned into a signal. If the weights were negative so that z = −1.7, ReLU would give 0 and Sigmoid about 0.154.

The flow is: Inputs → Weighted sum (z) → Activation f(z) → Output (a)

Why are linear transformations alone not enough?

Suppose we stack two linear transformations:

y = 2x + 1

z = 3y + 2

Substitute y into the second equation:

z = 3(2x + 1) + 2

z = 6x + 3 + 2

z = 6x + 5

The result is still a linear/affine transformation.

3. Important Activation Functions

Sigmoid
Tanh
ReLU
Leaky ReLU
Softmax

1. Sigmoid

Mathematical definition

Why it lands between 0 and 1: The term e^(−x) is always positive. When x is a large positive number, e^(−x) → 0, so σ → 1/(1+0) = 1. When x is a large negative number, e^(−x) → ∞, so σ → 1/∞ = 0. At x = 0, σ = 1/(1+1) = 0.5. The result is a smooth S-shaped curve. Derivativ

Behavior

Positive input: output moves closer to 1
Negative input: output moves closer to 0
Input = 0: output = 0.5

Intuition

Think of sigmoid as a probability-like function. It converts any number into a value between 0 and 1.

Advantages

Smooth and easy to understand
Useful for binary classification output
Output can be interpreted as probability

Limitations

Can cause vanishing gradients
Not zero-centered
Usually not preferred in hidden layers

Where used: output layer for binary classification and multi-label classification, and gates in LSTM/GRU. Rarely in hidden layers of modern deep nets.

2. Tanh

Mathematical definition

Behavior

If there id large positive input then output approches +1
If there id large negative input then output approches -1
If input is 0 then output is 0

Intuition

Tanh is similar to sigmoid, but instead of giving values from 0 to 1, it gives values from -1 to +1.

How it differs from Sigmoid

The key difference is that tanh is zero-centered. Because outputs can be negative as well as positive, the inputs to the next layer have mean closer to 0, which keeps weight gradients from being forced to share the same sign and generally makes optimization faster and more stable. Its larger peak gradient (1 vs 0.25) also means signals shrink less per layer in the non-saturated region.

Advantages

Zero-centered
Stronger than sigmoid for some hidden-layer tasks
Smooth function

Limitations

Can also suffer from vanishing gradients
Can become saturated for very large positive or negative inputs

Where used: hidden states in RNNs/LSTMs/GRUs (the candidate state and cell output), and in some shallow networks or GAN generator outputs where data is scaled to [−1, 1]

3. ReLU

Mathematical definition

Behavior

Positive input: passes the value directly
Negative input: becomes 0
Input = 0: output = 0

Intuition

ReLU basically says if output is positive keep it if negative make it zero

Why ReLU became the default in deep networks

No saturation for positive inputs. The gradient is a constant 1, so it does not shrink as it travels backward through many active layers. This largely fixes the vanishing gradient problem for the active path.
Cheap. Just a comparison, with no exponentials.
Sparsity. Roughly half of the neurons output exactly 0 for a given input, which creates sparse, efficient representations.
Faster convergence. AlexNet (2012) famously trained several times faster with ReLU than with tanh.

How it differs from Sigmoid and Tanh: those two squash both ends and have vanishing gradients on both sides. ReLU is unbounded on the positive side (no squashing, no saturation) and hard-zero on the negative side

Advantages

Very simple and fast
Helps reduce vanishing-gradient problems
Widely used in hidden layers

Limitations

Can suffer from the dying ReLU problem
Negative inputs always produce zero

Where used: default for hidden layers in CNNs, MLPs, and many deep architectures

4. Leaky ReLU

Mathematical definition

Behavior

Positive input: passes it normally
Negative input: keeps a small negative value instead of becoming zero

What it fixes

Standard ReLU gives zero gradient for negative z, so a neuron whose z is stuck negative can never recover. Leaky ReLU keeps a gradient of α there, which gives the weights a chance to update and move the neuron back into the active region.

Intuition

Leaky ReLU is basically ReLU with a small leak for negative values.

Advantages

Reduces the dying ReLU problem
Allows negative values to pass through
Simple and fast

Limitations

The negative slope α\alpha needs to be chosen
Not always better than standard ReLU

Where used: GAN discriminators (DCGAN popularized it), and deep networks where dying neurons are observed.

5. Softmax

Softmax is mainly used in the output layer for multi-class classification.

Mathematical definition

Output range:

0<f(xi)<1

Derivative

What are logits?

Logits are the raw, unnormalized scores produced by the final linear layer, one per class. They can be any real number (negative, positive, large, small) and are not probabilities. A logit is a score that says “how much does the network favor this class?

Behavior

Larger input → larger probability
Smaller input → smaller probability
Converts scores into probabilities

Intuition

Think of Softmax as a competition between classes.

Advantages

Produces probabilities
Probabilities add up to 1
Very useful for multi-class classification

Limitations

Mainly useful for classification outputs
Can be sensitive to very large values
Not normally used in hidden layers

What are logits?

Why exponentiation?

It makes every score strictly positive, which probabilities must be.

2. It is monotonic, so the ordering of classes is preserved: the largest logit still gets the largest probability. Page 8 of 10 Activation Functions: How Neural Networks Learn to Bend

Get Pothula Vineeth’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

3. It amplifies differences: a gap between logits becomes a multiplicative gap in probabilities. This makes the network “decisive” while staying smooth and differentiable (unlike a hard argmax).

4. It pairs neatly with cross-entropy loss: the combined gradient with respect to the logits becomes simply (p − y)

Why the outputs sum to 1.

Each output is a share of the total: Σᵢ P(y=i) = (Σᵢ e^(zᵢ)) / (Σⱼ e^(zⱼ)) = 1. The denominator is by construction the sum of all numerators, so the outputs always form a valid probability distribution.

Worked example. Logits z = [2.0, 1.0, 0.1]

e² = 7.389, e¹ = 2.718, e⁰.1 = 1.105. Sum = 11.212

Probabilities = [0.659, 0.242, 0.099]. They sum to 1.000.

Graph representation

4. The Vanishing Gradient Problem

Gradient tells the neural network how to change its weights to reduce the error.

What a gradient is and why we need it

A gradient is the derivative of the loss with respect to a parameter: ∂L/∂w. It answers “if I nudge this weight up a tiny bit, how much does the loss change, and in which direction?” Training is gradient descent:

w_new = w_old − η · ∂L/∂w        (η = learning rate)

If the gradient for a weight is near zero, the update is near zero, and that weight stops learning, however wrong it is.

How gradients travel backward

Backpropagation applies the chain rule layer by layer from the output back to the input. For a network with layers 1..L, where aₗ = f(zₗ) and zₗ = Wₗaₗ₋₁ + bₗ, the gradient reaching an early layer contains a product with one factor per layer in between:

∂L/∂a₁ = ∂L/∂a_L · [f'(z_L) · W_L] · [f'(z_(L−1)) · W_(L−1)] · ... · [f'(z₂) · W₂]

Each layer contributes a factor of f’(z) × weight. The activation derivative f’(z) is part of every single factor.

Why Sigmoid and Tanh produce tiny gradients

Sigmoid’s derivative never exceeds 0.25 (at z = 0) and drops toward 0 when |z| is large.
Tanh’s derivative peaks at 1 but is far below 1 for most inputs, and it also goes to 0 when saturated.

Saturation is the main culprit. If z = 6, then σ(6) ≈ 0.9975 and σ’(6) = 0.9975 × 0.0025 ≈ 0.0025. Almost no gradient flows through that neuron.

Repeated multiplication of small numbers

Suppose every layer’s sigmoid derivative is at its maximum, 0.25, and the weights are around 1. The gradient after passing back through n layers is roughly:

Layers passedGradient scale (0.25ⁿ)10.2530.015650.0009810about 10⁻⁶²⁰about 10⁻¹²

The real situation is usually worse, because most neurons aren’t at z = 0. The gradient shrinks exponentially with depth.

Effect on learning

The layers near the output receive healthy gradients and learn. The earliest layers, which should learn the basic building-block features, receive almost nothing. Their weights barely move, so the network trains very slowly or stalls, and the early layers stay close to random initialization. This is why deep sigmoid/tanh networks were so hard to train before around 2010.

Why ReLU helps

For z > 0, ReLU’(z) = 1. Multiplying by 1 neither shrinks nor grows the gradient, so a path of active ReLU neurons passes the gradient backward without activation-induced decay:

sigmoid, 10 layers: ~0.25^10 ≈ 1e-6      ReLU, 10 active layers: 1^10 = 1

Caveat: the weight factors still matter, so exploding or vanishing gradients can still come from W. That is why careful initialization (He initialization), normalization layers, and residual connections are used together with ReLU. Also, ReLU’s derivative is 0 for negative inputs, which leads to the next problem.

5. The Dying ReLU Problem

What is a dead neuron?

A dead neuron is a ReLU neuron whose pre-activation z is negative for essentially every input in the dataset. It outputs 0 all the time and contributes nothing to the network.

How does a neuron die?

A large gradient update (often from a too-high learning rate) pushes the weights or bias to a point where w·x + b < 0 for all typical inputs. A big negative bias is the classic cause.
Poor initialization or unusual input distributions can do the same.
Once z < 0 for all inputs, the output is 0.

Why does it stop learning?

The weight update depends on the gradient:

∂L/∂w = ∂L/∂a · f'(z) · x

For ReLU with z < 0, f’(z) = 0, so ∂L/∂w = 0. The weight update is exactly zero. Since the weights never change, z stays negative, so the neuron stays dead. There is no gradient signal to escape with, so the neuron is permanently stuck. In a badly tuned network a large share of neurons can die, wasting capacity.

How Leaky ReLU helps

With Leaky ReLU, f’(z) = α (for example 0.01) when z < 0 instead of 0. The gradient is small but not zero, so ∂L/∂w = ∂L/∂a · α · x ≠ 0. The weights can still change, and the neuron can drift back toward z > 0 and recover.

Simple example: negative input

Let z = −3, upstream gradient ∂L/∂a = 2, input x = 1.5, and α = 0.01.

ReLULeaky ReLU (α = 0.01)Output a = f(−3)0−0.03f’(−3)00.01∂L/∂w = 2 · f’ · 1.500.03Weight update (η = 0.1)nonew changes by 0.003 (a nudge)

The numbers are tiny, but that is the point: with ReLU there is nothing to work with, while with Leaky ReLU repeated small updates can rescue the neuron.

6. Activation Functions and Backpropagation

The learning loop

Forward pass → Loss → Gradient calculation (backward pass) → Weight update → repeat

Forward pass: compute z and a = f(z) for every layer and finally the prediction ŷ.
Loss: measure how wrong the prediction is, L(ŷ, y).
Backward pass: compute ∂L/∂w for every weight using the chain rule.
Update: w ← w − η ∂L/∂w.

Why the activation derivative matters

Consider one neuron with z = w·x + b, a = f(z), and a loss L that depends on a. By the chain rule:

∂L/∂w = (∂L/∂a) · (∂a/∂z) · (∂z/∂w)
      =  ∂L/∂a   ·  f'(z)  ·   x

∂L/∂a: how the loss changes with the neuron’s output (comes from later layers)
f’(z): the derivative of the activation function
∂z/∂w = x: the input to this weight

The factor f’(z) acts as a gate or volume knob on the learning signal. If f’(z) ≈ 0 (saturated sigmoid, negative-input ReLU), the weight update is crushed. If f’(z) = 1 (positive-input ReLU), the signal passes through at full strength.

Worked example (sigmoid output, squared error)

Let x = 1, w = 0.5, b = 0, target y = 1.

z = 0.5, a = σ(0.5) = 0.6225
L = ½(a − y)² = ½(−0.3775)² = 0.0713
∂L/∂a = a − y = −0.3775
σ’(z) = a(1 − a) = 0.6225 × 0.3775 = 0.2350
∂L/∂w = (−0.3775)(0.2350)(1) = −0.0887
Update with η = 0.5: w = 0.5 − 0.5(−0.0887) = 0.5444 (moves toward predicting 1)

Now imagine the neuron were saturated at z = 8. Then a ≈ 0.99966 and σ’(z) ≈ 0.000335. Even if the prediction were badly wrong, the gradient would be multiplied by 0.000335 and learning would nearly freeze. The activation’s derivative controls how fast the network can correct itself.

Through multiple layers

For the weights of layer l, the gradient is built from an error signal δ that is multiplied element-wise by f’(z) at every layer as it travels backward:

δₗ = (Wₗ₊₁ᵀ δₗ₊₁) ⊙ f'(zₗ)        ∂L/∂Wₗ = δₗ aₗ₋₁ᵀ

This is the exact mechanism behind both vanishing gradients (f’ < 1 repeatedly) and dead neurons (f’ = 0).

Takeaway

Activation functions do two jobs. In the forward pass they provide non-linearity. In the backward pass their derivatives decide how much learning signal reaches each weight. A good activation function is expressive and has a well-behaved derivative.

7. Choosing the Right Activation Function

ComponentCommon choiceWhyHidden layersReLU / Leaky ReLU / GELUCheap, non-saturating positive side (gradient = 1), sparse activationsBinary classification outputSigmoidOne output squashed into (0, 1), read as P(class = 1), pairs with binary cross-entropyMulti-class (single label) outputSoftmaxProbabilities over K mutually exclusive classes that sum to 1, pairs with categorical cross-entropyMulti-label outputSigmoid per outputEach label is an independent yes/no questionRegression outputLinear (none)The target can be any real number, and squashing would limit the range of predictions

Understanding the reasoning behind each choice

Hidden layers need non-linearity and healthy gradients, not a particular output meaning. Hidden activations are internal features, so nobody needs them to be probabilities. ReLU gives non-linearity at the lowest cost and does not saturate for positive values, so deep nets train well.
Output layers must match the form of the target. A probability must be in [0, 1] (Sigmoid). A distribution over exclusive classes must sum to 1 (Softmax). A house price can be any positive real number, so a bounded sigmoid would make large prices impossible to predict (Linear).
The loss function and the output activation are a pair. Sigmoid + binary cross-entropy and Softmax + categorical cross-entropy give the clean gradient (prediction − target), avoiding the slowdown that sigmoid + squared error suffers when the output saturates.
If the regression target is always positive, a ReLU or softplus on the output is sometimes used to enforce it.

Must every layer use the same activation?

No. Different layers play different roles, and mixing is normal. A typical classifier uses ReLU in all hidden layers and Softmax only in the output layer. An LSTM uses sigmoid for its gates and tanh for its candidate values inside the same cell. Transformers use GELU in feed-forward blocks and Softmax in attention. The practical rule: pick the hidden activation for trainability, and pick the output activation for the form of the answer.

8. Practical Experiment

The code below implements Sigmoid, Tanh, ReLU, Leaky ReLU (plus Softmax), plots each function and its derivative, and prints a few numbers that back up the observations. Run it in a notebook or script, then paste the generated figures into your blog (Figure 1: activation functions, Figure 2: derivatives, Figure 3: vanishing gradient).

Code

python

import numpy as np
import matplotlib.pyplot as plt

# ---------- 1. Activation functions ----------
def sigmoid(x):
    return 1 / (1 + np.exp(-x))def tanh(x):
    return np.tanh(x)def relu(x):
    return np.maximum(0, x)def leaky_relu(x, alpha=0.01):
    return np.where(x > 0, x, alpha * x)def softmax(z):
    z = z - np.max(z)            # subtract max for numerical stability
    e = np.exp(z)
    return e / e.sum()# ---------- 2. Derivatives ----------
def d_sigmoid(x):
    s = sigmoid(x)
    return s * (1 - s)def d_tanh(x):
    return 1 - np.tanh(x) ** 2def d_relu(x):
    return (x > 0).astype(float)def d_leaky_relu(x, alpha=0.01):
    return np.where(x > 0, 1.0, alpha)# ---------- 3. Input range ----------
x = np.linspace(-10, 10, 1000)funcs = [
    ("Sigmoid",    sigmoid,    d_sigmoid),
    ("Tanh",       tanh,       d_tanh),
    ("ReLU",       relu,       d_relu),
    ("Leaky ReLU", leaky_relu, d_leaky_relu),
]# ---------- 4. Figure 1: the functions ----------
fig, axes = plt.subplots(2, 2, figsize=(11, 8))
for ax, (name, f, df) in zip(axes.ravel(), funcs):
    ax.plot(x, f(x), linewidth=2)
    ax.axhline(0, color="gray", linewidth=0.8)
    ax.axvline(0, color="gray", linewidth=0.8)
    ax.set_title(f"{name} activation")
    ax.set_xlabel("Input z")
    ax.set_ylabel("Output f(z)")
    ax.grid(alpha=0.3)
plt.tight_layout()
plt.show()# ---------- 5. Figure 2: the derivatives ----------
fig, axes = plt.subplots(2, 2, figsize=(11, 8))
for ax, (name, f, df) in zip(axes.ravel(), funcs):
    ax.plot(x, df(x), color="crimson", linewidth=2)
    ax.axhline(0, color="gray", linewidth=0.8)
    ax.axvline(0, color="gray", linewidth=0.8)
    ax.set_title(f"Derivative of {name}")
    ax.set_xlabel("Input z")
    ax.set_ylabel("f'(z)")
    ax.grid(alpha=0.3)
plt.tight_layout()
plt.show()# ---------- 6. All functions on one plot ----------
plt.figure(figsize=(8, 5))
for name, f, _ in funcs:
    plt.plot(x, f(x), label=name, linewidth=2)
plt.ylim(-2, 4)
plt.title("Comparison of activation functions")
plt.xlabel("Input z")
plt.ylabel("Output f(z)")
plt.legend()
plt.grid(alpha=0.3)
plt.show()# ---------- 7. Numerical evidence ----------
for z in [-6, -1, 0, 1, 6]:
    print(f"z={z:>3}  sigmoid={sigmoid(z):.4f}  tanh={tanh(z):.4f}  "
          f"relu={relu(z):.2f}  leaky={leaky_relu(z):.2f}  |  "
          f"d_sig={d_sigmoid(z):.4f}  d_tanh={d_tanh(z):.4f}  "
          f"d_relu={d_relu(np.array(z)):.0f}  d_leaky={d_leaky_relu(z):.2f}")print("Softmax([2.0, 1.0, 0.1]) =", softmax(np.array([2.0, 1.0, 0.1])).round(3))# ---------- 8. Vanishing gradient demo (Figure 3) ----------
depth = np.arange(1, 21)
plt.figure(figsize=(8, 5))
plt.semilogy(depth, 0.25 ** depth, label="Sigmoid (best case, 0.25 per layer)")
plt.semilogy(depth, 1.0 ** depth, label="ReLU (active path, 1 per layer)")
plt.title("Gradient scale after passing back through n layers")
plt.xlabel("Number of layers")
plt.ylabel("Gradient scale (log axis)")
plt.legend()
plt.grid(alpha=0.3)
plt.show()

The printed table should show, for example, sigmoid(6) ≈ 0.9975 with derivative ≈ 0.0025, tanh(−6) ≈ −1.0000 with derivative ≈ 0.0000, ReLU(−6) = 0 with derivative 0, and Leaky ReLU(−6) = −0.06 with derivative 0.01.

activation functions

Observations from the graphs

Output range. Sigmoid stays inside (0, 1) and Tanh inside (−1, 1), so both are bounded. ReLU is [0, ∞) and Leaky ReLU is (−∞, ∞), so both are unbounded above, and the plots keep climbing along y = x for positive inputs.

Saturation. The Sigmoid and Tanh curves flatten into horizontal lines beyond roughly |z| > 4 to 5. In that flat region, large changes in z barely change the output, which is saturation. ReLU and Leaky ReLU never flatten on the positive side. ReLU is flat only for negative z (output 0), while Leaky ReLU keeps a faint slope there.

Zero-centered behavior. Tanh passes through the origin and is symmetric (an odd function): tanh(−z) = −tanh(z). Sigmoid crosses at (0, 0.5), so all its outputs are positive and its average output is around 0.5, not 0. ReLU outputs are all ≥ 0, so it is not zero-centered. Leaky ReLU’s small negative values make it only slightly closer to zero-centered.

Gradient behavior (Figure 2). The Sigmoid derivative is a narrow bell with peak 0.25, nearly zero for |z| > 5. The Tanh derivative is a taller bell with peak 1, also nearly zero in the tails. These two curves are the visual form of the vanishing gradient problem. The ReLU derivative is a step: exactly 0 for z < 0 and exactly 1 for z > 0, so it never decays on the positive side but is dead on the negative side. Leaky ReLU’s derivative steps from 0.01 to 1, so it is never exactly zero.

Positive vs negative inputs. For positive z, ReLU and Leaky ReLU are identical (the identity function), while Sigmoid and Tanh compress. For negative z, the four behave very differently: Sigmoid heads to 0, Tanh to −1, ReLU is exactly 0, and Leaky ReLU is a small negative number growing linearly. Figure 3 shows 0.25ⁿ collapsing exponentially against the flat line of 1ⁿ for ReLU.

9. Activation Function Comparison

FunctionOutput rangeZero-centered?Main advantageMain limitationTypical useSigmoid(0, 1)No (outputs always positive, centered around 0.5)Smooth, probability-like outputSaturates at both ends, so gradients vanish (max derivative 0.25)Binary / multi-label output layers, LSTM/GRU gatesTanh(−1, 1)YesZero-centered with a stronger gradient (max 1) than sigmoidStill saturates in the tails, so gradients vanishRecurrent network states, some shallow networksReLU[0, ∞)NoCheap, no positive-side saturation, gradient is 1, sparse activationsDying ReLU: zero gradient for z < 0Default hidden layer in CNNs and MLPsLeaky ReLU(−∞, ∞)Approximately (slightly better than ReLU, not truly centered)Keeps ReLU’s benefits and a non-zero gradient for z < 0Slope α is a hyperparameter, with a modest benefit in practiceHidden layers prone to dead neurons, GAN discriminatorsSoftmax(0, 1) per output, sum = 1No (outputs are positive)Turns logits into a probability distribution over exclusive classesOutput layers only, can be overconfident, needs numerical care with large logitsMulti-class classification output, Transformer attention weights

Every row follows from the explanations above: the ranges come from the formulas in Section 3, the vanishing-gradient entries from the derivative bounds in Section 4, and the dying-ReLU entry from Section 5.

10. Conclusion

Why activation functions are necessary. Without them a neural network is a stack of linear maps, and a stack of linear maps collapses into one linear map (W₂W₁ = W’). Depth adds nothing, and the model can only draw straight boundaries. An activation function breaks that collapse.

Why non-linearity matters. Non-linear functions let each layer fold and bend the data space instead of just rotating and scaling it. XOR is the smallest example: unsolvable linearly, trivial after one non-linear hidden layer. In general, non-linearity is what lets networks build curved decision boundaries and hierarchical features, and it is what the universal approximation theorem relies on.

How activations affect learning. The activation matters twice. Going forward it shapes what the network can represent. Going backward, its derivative f’(z) appears in every term of the chain rule, so it controls how much learning signal reaches each weight. Saturating functions shrink that signal, a zero derivative blocks it, and a derivative of 1 passes it intact.

Why ReLU won. ReLU is cheap, it does not saturate for positive inputs (gradient exactly 1), and it produces sparse activations. Together these let much deeper networks train faster than with sigmoid or tanh, which is why it became the hidden-layer default.

The limitations that remain. Sigmoid and Tanh saturate and cause vanishing gradients. ReLU can leave neurons dead, and it is not zero-centered. Leaky ReLU softens the dying problem but adds a hyperparameter. Softmax can be overconfident and is tied to the output layer. No activation is perfect, and each one trades something for something else.

Why different tasks need different output activations. The output must live in the same space as the target. Probabilities need (0, 1), exclusive class distributions need to sum to 1, and unbounded real values need no squashing. So hidden layers are chosen for trainability and output layers for the meaning of the answer.

The bigger answer to “how do activation functions influence what a network can learn and how well it learns it?” is this: the shape of the function sets what the network can express, and its derivative sets how efficiently the network can discover that expression through backpropagation.

BONUS: GELU (Gaussian Error Linear Unit)

Mathematical formulation

GELU(x) = x · Φ(x)

where Φ(x) is the cumulative distribution function of the standard normal distribution. A widely used approximation is:

GELU(x) ≈ 0.5 · x · (1 + tanh[ √(2/π) · (x + 0.044715 x³) ])

Motivation

ReLU makes a hard, deterministic decision: keep x if x > 0, otherwise drop it. GELU makes a soft, probabilistic one. It multiplies the input by the probability Φ(x) that a standard normal variable is less than x. Large positive inputs get a weight near 1 (kept), large negative inputs get a weight near 0 (dropped), and values near zero are partially kept. It was introduced (Hendrycks & Gimpel, 2016) as a smooth alternative that blends ideas from ReLU and dropout.

How it differs from ReLU

Smooth: no kink at 0, so the derivative is continuous.
Non-monotonic: for slightly negative inputs (around x ≈ −0.75) the output dips to a small negative minimum (about −0.17) before returning toward 0, instead of being exactly 0.
Non-zero gradient for moderately negative inputs, which reduces the dying-neuron problem. For very negative x, both output and gradient go to about 0.

Advantages

Smooth gradients, which tend to help optimization in very deep models
Often gives slightly better accuracy than ReLU in Transformer-type models
Lets small negative values through, which carries more information than a hard cut-off

Limitations

More expensive than ReLU (involves erf or tanh)
Not sparse in the way ReLU is, since few outputs are exactly 0
Benefits over ReLU are task-dependent and sometimes small

Where it is commonly used

Transformer architectures such as BERT and the GPT family use GELU in their feed-forward blocks, and many vision transformers do too.

Why choose it over ReLU?

Choose GELU when you are building or fine-tuning a Transformer, where it is the standard and pretrained weights expect it, or when you want smoother gradients and can afford slightly more computation. Stay with ReLU for plain CNNs or small models where speed and simplicity matter most.

Code: GELU vs ReLU graph

python

from scipy.special import erf

def gelu(x):
    return 0.5 * x * (1 + erf(x / np.sqrt(2)))x = np.linspace(-5, 5, 1000)
plt.figure(figsize=(8, 5))
plt.plot(x, relu(x), label="ReLU", linewidth=2)
plt.plot(x, gelu(x), label="GELU", linewidth=2)
plt.axhline(0, color="gray", linewidth=0.8)
plt.axvline(0, color="gray", linewidth=0.8)
plt.title("GELU vs ReLU")
plt.xlabel("Input z")
plt.ylabel("Output f(z)")
plt.legend()
plt.grid(alpha=0.3)
plt.show()

What the graph shows: for large positive x the two curves almost coincide (GELU ≈ x). Near zero, ReLU has a sharp corner while GELU rounds it off smoothly. For negative x, ReLU is exactly 0, while GELU dips slightly below zero and then returns to 0.