An activation function is a mathematical function applied to a neuron’s weighted sum of inputs to decide the neuron’s output. Its most important job is to add non-linearity. Without it, a neural network of any depth would behave like a single straight-line model and could not learn complex patterns such as faces, speech or language.
Key takeaways
- Every artificial neuron computes a weighted sum, then applies an activation function.
- Non-linear activations let stacked layers model curves and complex boundaries.
- ReLU is the default for most hidden layers.
- Sigmoid and softmax are used in output layers to produce probabilities.
- GELU and similar smooth functions are common in transformer models.
An activation function explained in simple terms
Think of a dimmer switch. The electricity flowing in is the neuron’s weighted sum. The dimmer decides how much light comes out. Some dimmers stay off until a threshold, then brighten steadily (ReLU). Others brighten gradually and level off at full brightness (sigmoid). The shape of the dimmer changes how the whole lighting system behaves, and a network full of well-chosen dimmers can create very rich patterns.
Why non-linearity matters
If every layer only multiplies and adds, stacking layers just produces another multiplication and addition. Ten linear layers collapse into one. That network could only separate data with a straight line, which is why a single neuron cannot solve the XOR problem. A non-linear activation between layers breaks this collapse, letting the network bend its decision boundaries. With enough neurons and a suitable activation, a network can approximate almost any continuous function, a result known as the universal approximation theorem.
Common activation functions
| Function | Output range | Where used | Notes |
|---|---|---|---|
| ReLU | 0 to ∞ | Hidden layers (default) | Simple, fast; outputs 0 for negative input |
| Leaky ReLU | −∞ to ∞ | Hidden layers | Small slope for negatives to avoid “dead” neurons |
| Sigmoid | 0 to 1 | Binary output, gates in LSTM and GRU | Can cause vanishing gradients in deep stacks |
| Tanh | −1 to 1 | RNN hidden states | Zero-centred, still saturates at extremes |
| Softmax | 0 to 1, sums to 1 | Multi-class output layer | Turns scores into class probabilities |
| GELU | about −0.17 to ∞ | Transformers | Smooth version of ReLU; introduced in 2016 |
| SiLU / Swish | about −0.28 to ∞ | Modern CNNs and LLMs | Smooth, performs well in deep networks |
A worked example
A neuron’s weighted sum for three different inputs is −2, 0.5 and 3.
| Weighted sum | ReLU | Sigmoid | Tanh |
|---|---|---|---|
| −2 | 0 | 0.12 | −0.96 |
| 0.5 | 0.5 | 0.62 | 0.46 |
| 3 | 3 | 0.95 | 0.995 |
ReLU passes positive signals unchanged and blocks negatives. Sigmoid squeezes everything into a probability-like range. Tanh squeezes into −1 to 1. Notice sigmoid and tanh flatten out at large values; there the gradient is tiny, which slows learning in deep networks.
For softmax, suppose an output layer produces scores 2.0, 1.0 and 0.1 for “cat”, “dog” and “bird”. Softmax converts them to roughly 0.66, 0.24 and 0.10, probabilities that add to 1.
Why ReLU became the default
Before around 2010, sigmoid and tanh were common, but they caused vanishing gradients in deep networks. ReLU, promoted for deep networks by Vinod Nair and Geoffrey Hinton in 2010 and used in AlexNet in 2012, has a constant gradient for positive inputs, trains faster and is cheap to compute. Its weakness is “dying ReLU”: neurons that always receive negative input output zero and stop learning. Leaky ReLU and GELU reduce this problem.
How to choose an activation function
- Hidden layers: start with ReLU; try GELU or SiLU for transformers and deep models.
- Binary classification output: sigmoid.
- Multi-class output: softmax.
- Multi-label output: sigmoid on each label.
- Regression output: usually no activation (linear).
Limitations
- Saturating functions (sigmoid, tanh) can slow learning.
- ReLU neurons can die and stop updating.
- The best choice often depends on architecture and needs testing.
- Smooth functions such as GELU cost slightly more to compute.