Weights: What the Model Is Actually Learning
My last post was about optimization, and it kept coming back to the weights: gradient descent updates them, the optimizer searches for better ones. But I never stopped to ask where they come from or what they turn into in a big model. So I went back to that question. (I first learned the training mechanics from Professor Rama’s MIT deep learning course.)
A weight is a number that scales an input
In the optimization post I wrote ŷ = wx + b, where ŷ is the prediction, the weight is the slope, and the bias is the intercept. With more inputs, each one gets its own weight. A house price model might look like this:
ŷ = w₁ × square footage + w₂ × bedrooms + w₃ × age + b
Here b is the bias, a starting offset. In a simple model like this, a bigger weight means each extra unit of that input changes the prediction more, and a negative weight pushes it the other way. Nobody types these numbers in. The model has to find them from data.
A neural network repeats this idea many times. Each neuron multiplies its inputs by weights, adds a bias, and passes the result through an activation function (I wrote about those here). When people say a model has billions of parameters, they mostly mean weights and biases.
Where do the weights come from?
Before training, the model knows nothing, so the weights start as small random numbers, with the scale chosen carefully for each layer.
Why random? If every neuron in a layer started with the same weights, they would all produce the same output and get the same update. They would stay identical, and the layer would behave like one neuron. Random starting values let each neuron learn something different.
How they change
Get Fahimeh Pouryani’s stories in your inbox
Join Medium for free to get updates from this writer.
The loop is the one from my last post: predict, measure the loss (how wrong the prediction was), adjust, repeat.
In its simplest form, the adjustment is:
new weight = old weight − learning rate × gradient
The learning rate is the step size. The gradient says how the loss changes if that weight changes. If the gradient is positive (raising the weight would raise the loss), the weight goes down. If it is negative, the weight goes up. For example, with a weight of 0.42, a gradient of 0.15, and a learning rate of 0.1, the new weight is 0.42 − 0.1 × 0.15 = 0.405.
That’s what made optimization click for me: it isn’t something that happens next to the model. The weights are what’s being changed.
Weights in a language model
Large language models (LLMs), the kind behind AI chatbots, build on the same idea at a much bigger scale. They have billions of parameters, mostly weights. During training, a model reads text and learns to predict the next word, roughly speaking. Given “The capital of France is”, it gives each possible next word a probability. The right answer is whatever actually came next in the training text. If that word gets a low probability, the loss is high and the weights get adjusted. This repeats over an enormous amount of text.
There’s no database of sentences inside an LLM. What it picked up is encoded in the weights, and you can’t point to one weight and say what it knows. The weights also only work together with the architecture that uses them. Still, they are the result of all that training, which is why releasing a model’s weights is a big deal. It isn’t just code, and others can build on it instead of starting from random numbers.
Where this leaves me
That leaves a question I’ve been avoiding. The gradient tells each weight which way to move, but a model has billions of weights, many of them buried layers deep. How is the gradient worked out for every one of them? That’s backpropagation, and it’s the next post.
