What is a Weight in Neural Networks?

A weight is a learned number in a neural network that controls how strongly one input or neuron influences the next neuron. Each connection between neurons has its own weight. Positive weights amplify a signal, negative weights reverse it, and weights near zero mostly ignore it. Together, the weights store everything the network has learned, which is why a model’s size is often described by its number of parameters.

Key takeaways

• Weights and biases together are called a model’s parameters.
• Training adjusts weights using backpropagation and gradient descent.
• Weights are initialised with small random values, not zeros.
• Large language models have billions of weights.
• “Open-weight” models publish their trained weights so others can run or fine-tune them.

A weight explained in simple terms

Imagine a recruiter scoring job applicants on experience, skills test and interview. If they multiply experience by 0.5, test by 0.3 and interview by 0.2, those multipliers are weights: they reflect how much each factor matters. A neural network learns its own multipliers from data, for millions of connections at once, rather than having a person set them.

Where weights sit

In a single artificial neuron: output = activation(w₁x₁ + w₂x₂ + … + wₙxₙ + b). Each w is a weight. In a full layer, the weights form a matrix: one row per neuron, one column per input. A dense layer with 100 inputs and 50 neurons has 100 × 50 = 5,000 weights, plus 50 biases.

A worked example: how weights change

A neuron predicts house price (in lakhs) from size in hundreds of square feet. Start with weight w = 2 and bias 0. A house of size 10 actually sold for 60.

• Prediction: 2 × 10 = 20. Error: 20 − 60 = −40.
• The gradient says increasing w would reduce the error. With a small learning rate, w moves from 2 to, say, 3.6.
• New prediction: 3.6 × 10 = 36. Closer.

Across thousands of houses and many rounds, w settles near the value that best fits the data. Now imagine this happening for billions of weights at once: that is training a large model.

How weights are initialised

If all weights start at zero, every neuron in a layer computes the same thing and they never become different. So weights start with small random values. The scale matters: too large and signals explode through layers; too small and they vanish. Common schemes include Xavier (Glorot) initialisation (2010), suited to sigmoid and tanh, and He initialisation (2015), suited to ReLU.

Counting weights

More weights mean more capacity to learn, but also more memory, compute and data needed. A model with 7 billion weights stored in 16-bit precision needs roughly 14 GB just to hold them, which is why quantisation (storing weights with fewer bits) matters for running models on smaller hardware.

Weights in practice

• Weight decay (L2 regularization) keeps weights small to reduce overfitting.
• Fine-tuning adjusts pre-trained weights for a new task. Methods like LoRA add small trainable weight updates instead of changing all weights.
• Open-weight models release their weights publicly. This is different from fully open-source models, which also release training code and, ideally, data.
• Weight files are what you download when you run a model locally.

Weights vs biases vs hyperparameters

Limitations

• Individual weights are rarely interpretable on their own.
• Huge weight counts require expensive hardware.
• Poor initialisation can stall training.
• Trained weights can encode bias from training data, and can sometimes leak memorised training examples.