In a neural network, a bias is a learned number added to a neuron’s weighted sum before the activation function is applied. It shifts the neuron’s output up or down, independent of the inputs. This lets the neuron activate at the right threshold and lets the network fit patterns that do not pass through zero. Each neuron usually has one bias.
This page covers the bias term, a model parameter. It is different from model bias, which refers to systematic prediction error or unfair outcomes.
Key takeaways
- Neuron output = activation(weighted inputs + bias).
- The bias shifts the decision threshold; the weights set the slope or direction.
- Biases are learned during training, just like weights.
- Biases are usually initialised to zero or a small constant.
- The bias term has nothing to do with fairness bias or statistical bias.
A bias explained in simple terms
Think of the equation of a line: y = mx + c. The slope m is like a weight. The intercept c is like a bias. Without c, every line must pass through the origin, which is very limiting. A line for “taxi fare = rate × distance + base fare” needs the base fare, because even a zero-kilometre ride costs something. The bias is the neuron’s base fare.
A worked example: the threshold shift
A neuron uses a step activation (outputs 1 if the sum is above 0) to decide whether to approve a loan based on one score x, with weight 1.
- With bias 0: the neuron approves anyone with a score above 0.
- With bias −650: sum = x − 650, so it approves only scores above 650.
- With bias −750: it approves only scores above 750.
The weight did not change. The bias alone moved the approval threshold. In a real network, training finds the best bias automatically.
For another view, take the AND and OR example: with weights 1 and 1, a bias of −1.5 makes the neuron compute AND, while a bias of −0.5 makes it compute OR.
Why biases matter
- Flexibility: neurons can activate even when all inputs are zero, or stay off unless inputs are large.
- Better fit: without biases, every decision boundary would have to pass through the origin, which rarely matches real data.
- Activation positioning: biases shift where the “bend” of a ReLU or the steep part of a sigmoid happens, letting each neuron respond in its most useful range.
How biases are trained and stored
Biases are updated by backpropagation and gradient descent alongside weights. There are far fewer biases than weights: a dense layer with 100 inputs and 50 neurons has 5,000 weights but only 50 biases. They are commonly initialised to zero, because the random weights already make neurons different from each other. Some layers leave out biases, for example when a normalisation layer that follows has its own shift parameter, which would make a separate bias redundant.
Three meanings of “bias” in AI
| Term | Meaning | Example |
|---|---|---|
| Bias term (this page) | Learned offset parameter in a neuron | Shifts a neuron’s activation threshold |
| Statistical model bias | Systematic error from overly simple assumptions | Linear model underfitting curved data |
| Fairness bias | Unfair or discriminatory outcomes for groups | A hiring model that disadvantages a group |
Mixing these up causes confusion, especially in content about AI ethics. When writing or reading about “bias”, check which meaning is intended.
Weights vs biases
| Aspect | Weight | Bias |
|---|---|---|
| Attached to | Each connection | Each neuron |
| Role | Scales an input’s influence | Shifts the neuron’s output |
| Count | Very many | Relatively few |
| Typical initialisation | Small random values | Zero or small constant |
| Line analogy | Slope | Intercept |
Limitations
- On its own, a bias cannot capture relationships between inputs; that is the weights’ role.
- Redundant biases waste parameters when normalisation layers already shift outputs.
- The word “bias” is easily confused with fairness bias in non-technical discussions.