Activation Functions

↑ Top

Introduction

An activation function defines how a neuron transforms its inputs before passing them forward. By introducing non‑linearity, these functions give neural networks the expressive power to model complex relationships. This article examines their role, evolution from step functions to ReLU and beyond, and the properties that make some more effective than others.

The Goal

By the end of this article, you will understand:

• What activation functions do and why they’re essential.
• Why the step function is too limited for modern networks.
• How common activations (sigmoid, tanh, ReLU, Mish) work and their trade‑offs.
• How non‑linearity enables networks to build complex, multi‑layer representations.

Activation Functions Explained

At its core, an activation function decides how much of a neuron’s input “makes it through” to the next layer.

The perceptron’s step function acts like a binary switch — fully on or fully off — whereas modern activation functions allow graded responses.

This ability to represent varying levels of activation is what makes neural networks so powerful. Instead of restricting neurons to binary outputs, activation functions enable networks to represent more nuanced relationships.

Why Neural Networks Need Non‑Linear Activation

Without non‑linear activations, stacking layers adds no new representational power: the network collapses to a single linear transformation. Non‑linear functions break this limitation, letting each layer reshape inputs in new ways and enabling flexible decision boundaries.

For example, stacking two linear layers:

$$\begin{aligned} y &= W_2 (W_1 x + b_1) + b_2 \\ &= (W_2 W_1) x + (W_2 b_1 + b_2) \end{aligned}$$

This simplifies to just one linear transformation — offering no additional representational power compared to a single layer.

Non‑linear activations break this limitation. They allow each layer to reshape the data space in new ways, enabling the network to learn curved, flexible decision boundaries instead of being limited to straight lines or flat planes.

The Step Function

In the previous article, we introduced the step function — Rosenblatt’s original choice for the perceptron. It outputs either 0 or 1 based on whether the combined input $z$ crosses a threshold:

$$f(z) = \begin{cases} 1 & \text{if } z \geq \theta \\ 0 & \text{if } z < \theta \end{cases}$$

These weaknesses — lack of gradient, hard switching, and only linear boundaries — make it unsuitable for training deep networks. Researchers quickly replaced it with smoother, differentiable functions that could support multi‑layer learning.

Common Activation Functions

Modern activation functions address the limitations of Rosenblatt’s step function. They provide gradients for learning, handle a wider range of inputs, and give networks the expressive power needed for complex tasks.

Each one reshapes inputs differently, offering distinct trade-offs in training speed, stability, and representational power.

---

Sigmoid

Definition:
The sigmoid function maps any real-valued input into a smooth curve between 0 and 1:

$$f(z) = \frac{1}{1 + e^{-z}}$$

Intuition:
Acts like a soft switch: small inputs approach 0, large inputs saturate near 1, and the transition between them is smooth.

Pros:

• Differentiable everywhere (supports backpropagation).
• Outputs between 0 and 1, making it interpretable as probabilities.

Cons:

• Vanishing gradients: Extreme inputs flatten the curve, slowing or halting learning.
• Not zero-centered, which can make optimization less efficient.

Use cases:

• Still common in output layers for binary classification tasks.
• Rarely used in hidden layers of deep networks (mostly replaced by ReLU).

---

Tanh

Definition:
The hyperbolic tangent (tanh) function is similar to sigmoid but outputs between -1 and 1, making it zero-centered:

$$f(z) = \tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$$

Intuition:
Like sigmoid, but centered at zero, allowing for both negative and positive outputs.

Pros:

• Zero-centered: Keeps activations balanced, improving gradient flow.
• Smooth and differentiable.

Cons:

• Suffers from the same vanishing gradient issue at extreme values.

Use cases:

• Common in hidden layers for shallow networks.
• Still used in recurrent neural networks (RNNs).

---

ReLU

Definition:
The Rectified Linear Unit (ReLU) outputs zero for negative inputs and passes positive inputs through unchanged:

$$f(z) = \max(0, z)$$

Intuition:
Like a one-way gate: negative inputs are blocked, positive inputs flow through.

Pros:

• Computationally cheap and simple to implement.
• No saturation for positive inputs → avoids vanishing gradients.
• Works well in deep networks, enabling faster training.

Cons:

• Dying ReLU problem: Neurons can become permanently inactive.
• Not zero-centered.

Use cases:

• Default activation for hidden layers in CNNs, MLPs, and transformers.

---

Leaky ReLU

Definition:
Leaky ReLU modifies ReLU by adding a small negative slope for $z < 0$:

$$f(z)=\begin{cases}z & z\ge0 \\ \alpha z & z<0\end{cases}$$

Intuition:
Allows small negative outputs instead of zero, helping prevent neurons from dying.

Pros:

• Fixes the “dying ReLU” problem.
• Still fast and efficient.

Cons:

• Introduces a hyperparameter $\alpha$.
• Not zero-centered.

Use cases:

• Preferred when training deep networks with dead neuron issues.

---

ELU

Definition:
The Exponential Linear Unit (ELU) smoothly blends negative inputs toward a small negative value:

$$f(z)=\begin{cases}z & z\ge0 \\ \alpha(e^z - 1) & z<0\end{cases}$$

Intuition:
By curving the negative side, ELU avoids dead neurons and makes activations zero-centered, improving learning dynamics.

Pros:

• Zero-centered output.
• Smooth gradient for negative inputs.

Cons:

• Slightly more expensive than ReLU.
• Still requires tuning $\alpha$.

Use cases:

• Deep networks where stable convergence is critical.

---

Mish

Definition:
The Mish activation is a smooth, non-monotonic function:

$$f(z) = z \cdot \tanh(\ln(1 + e^z))$$

Intuition:
Mish behaves like a smooth ReLU variant with better gradient flow and zero-centered output. It has been shown to improve performance in some deep models.

Pros:

• Smooth, zero-centered, and non-monotonic.
• Empirically effective in some vision/NLP tasks.

Cons:

• More computationally expensive.
• Still relatively new and less widely supported.

Use cases:

• Experimental models that benefit from smooth gradients.

---

Comparing Activation Functions

Here’s a quick reference comparing the activation functions we’ve covered — their output ranges, whether they’re zero‑centered, and why these details matter in practice.

Function	Output Range	Zero‑Centered?	Why It Matters
Step	0 or 1	No	Historically important but too limited for modern learning.
Sigmoid	0 to 1	No	Smooth, interpretable as probabilities, but saturates (vanishing gradients).
Tanh	‑1 to 1	Yes	Centered at zero for balanced outputs, but still suffers from vanishing gradients.
ReLU	0 to ∞	No	Simple, fast, and dominant in deep networks, though neurons can “die.”
Leaky ReLU	~‑∞ to ∞	No	Fixes the “dying ReLU” problem by allowing small negative outputs.
ELU	~‑1 to ∞	Yes	Improves convergence by smoothing negative values, but slower to compute.
Mish	~‑∞ to ∞	Yes	Smooth and promising for stability, though slower and less tested than ReLU.

---

Each of these activations represents a step forward from the binary switch of Rosenblatt’s step function — adding the flexibility, stability, and expressiveness needed for modern deep learning systems.

From Single‑Layer to Multi‑Layer Networks

In the Why Neural Networks Need Non‑Linear Activation section, we saw visually that stacking purely linear layers collapses into a single straight line or flat plane.

Now let’s make that intuition concrete with some math — and see why non‑linear activations are what make multi‑layer networks powerful.

---

From One Layer to Multiple Layers

Recall from the previous article that a single perceptron computes:

$$a = f(\mathbf{w}^\top \mathbf{x} + b)$$

This works well for classifying simple, linearly separable data.

But what if we want to model more complex relationships?

The natural idea is to stack perceptrons into layers — so the output of one layer becomes the input to the next.

If we stack two perceptrons without an activation function (i.e., using only linear transformations):

$$\begin{aligned} a^{(1)} &= W^{(1)} \mathbf{x} + b^{(1)} \\ a^{(2)} &= W^{(2)} a^{(1)} + b^{(2)} \\ &= W^{(2)}(W^{(1)} \mathbf{x} + b^{(1)}) + b^{(2)} \\ &= (W^{(2)} W^{(1)}) \mathbf{x} + (W^{(2)} b^{(1)} + b^{(2)}) \end{aligned}$$

This simplifies to a single linear transformation.

In other words: stacking linear layers gives you… one linear layer.

No matter how many you add, you can’t model anything more complex than a straight line or flat plane.

---

How Non‑Linear Activation Unlocks Complexity

Now insert a non‑linear activation function $f$ between the layers:

$$a^{(1)} = f(W^{(1)} \mathbf{x} + b^{(1)})$$ $$a^{(2)} = f(W^{(2)} a^{(1)} + b^{(2)})$$

This breaks the collapse.

Each layer can now reshape the data in a new, non‑linear way, making it possible for the network to model curved, complex decision boundaries and capture intricate patterns.

From Activation Functions to Multi‑Layer Networks

We’ve now seen how activation functions transform simple perceptrons into powerful building blocks for deep learning.

They introduce non‑linearity, enabling networks to model complex decision boundaries, and their design choices directly affect training stability, learning speed, and model performance.

These functions form the backbone of multi‑layer perceptrons (MLPs) — networks that stack perceptrons into layers to build increasingly rich representations of data.

In the next article, we’ll dive into multi‑layer perceptrons and explore backpropagation — the algorithm that lets these networks learn by adjusting weights across all layers.