Backpropagation
From vanishing gradients and NETtalk to the architecture of modern transformers
Introduction
Backpropagation became the dominant training algorithm for neural networks following a landmark 1986 paper by Rumelhart, Hinton, and Williams. The algorithm provided a principled method for adjusting the weights of a multilayer network based on the error at its output, making it possible to train networks with hidden layers for the first time on problems of practical complexity. The decades that followed revealed two limitations. The algorithm failed in deep networks because gradients diminished exponentially as they propagated backward through many layers, leaving early weights unchanged during training. The learning mechanism it uses has no known equivalent in biological neural systems, raising a question about whether the brain adjusts its connections through a fundamentally different process. This post traces how these two problems shaped the architecture of modern neural networks through a set of core questions.
What does backpropagation actually compute?
Understanding what backpropagation computes requires starting from what training is trying to achieve. Training a neural network means finding weights that minimize the difference between the network’s output and the correct output on a set of training examples. This difference is measured by a loss function, a scalar value that is large when the network is wrong and small when it is right. The goal of training is to move the weights in a direction that reduces the loss.
Backpropagation computes the gradient of the loss with respect to every weight in the network. A gradient is a collection of partial derivatives, one for each weight, where each partial derivative measures how much the loss would change if that particular weight changed by a small amount. With the gradient in hand, gradient descent updates every weight simultaneously by moving each one a small step in the direction that reduces the loss.
A neural network is a composition of functions, and computing how the loss depends on a weight in an early layer requires tracing a chain of dependencies from the output back through every intermediate layer. The output of the final layer depends on the weights of that layer and on the activations coming from the layer before it, which themselves depend on the weights of that layer and on the layer before it, and so on back to the input. Backpropagation handles this efficiently using the chain rule of calculus, which states that the derivative of a composed function is the product of the derivatives of each function in the chain. One forward pass computes the activations at every layer. One backward pass, propagating the gradient from the output layer to the input layer, computes the gradient with respect to every weight in a single sweep.
Why do gradients vanish in deep networks?
The chain rule multiplies derivatives together, and in a deep network the backward pass accumulates a product with one term per layer. Whether that product remains large enough to update early layers depends entirely on the magnitude of the individual terms.
The activation function used in early neural networks was the sigmoid. The sigmoid maps any real number to a value between zero and one, making it attractive as a model of neural firing rates. It was also a natural component in probabilistic systems like the Boltzmann machine, where outputs needed to represent probabilities. The derivative of the sigmoid reaches a maximum of 0.25 at the center of its input range and falls toward zero at both extremes. When the input to a sigmoid unit is large in magnitude, the unit is saturated, meaning it operates in a region where a large change in input produces almost no change in output, and its derivative is near zero.
Multiplying together many sigmoid derivatives, each of which is at most 0.25, produces a number that shrinks exponentially with the depth of the network. In a ten-layer network, a product of ten sigmoid derivatives in the saturating regime falls toward 0.25¹⁰ ≈ 0.000001. By the time the backward pass reaches the early layers, the gradient has vanished, and those layers receive an update signal so small that their weights change negligibly. The same mathematical property of the sigmoid that made it useful for probabilistic updates in the Boltzmann machine, its smooth and bounded output, is the property that makes it destructive in the backward pass of a deep network.
What architectural changes allow gradients to flow through deep networks?
The vanishing gradient problem was not solved by a single insight. A sequence of independent architectural innovations each addressed a specific aspect of the problem, and together they made deep networks trainable. Those innovations are the rectified linear unit (ReLU), careful weight initialization, batch normalization, and residual connections. Each one is present in the architecture of every transformer built today.
The most direct fix was replacing the sigmoid with the ReLU in the hidden layers of the network. The ReLU passes positive inputs through unchanged and sets negative inputs to zero, so its derivative is either zero or one, with no saturation for positive inputs, meaning the gradient passes through any active unit without shrinking. The practical effect was substantial. Networks that failed to train with sigmoid activations could be trained with ReLU activations with no other changes. ReLU units receiving negative inputs contribute nothing to the backward pass, which can cause some units to permanently stop updating, a problem known as dying ReLU, where a unit stops firing entirely and never recovers because the gradient through it is always zero. This turned out to be manageable in practice. Transformers use a smoother variant called GELU, the Gaussian Error Linear Unit, which avoids the hard zeroing of negative inputs but preserves the core property of an activation function whose derivative does not saturate.
Careful weight initialization addresses a related but distinct problem that appears before training even begins. The initial weights determine the initial activations, which determine whether the first backward pass produces useful gradients. Weights initialized too large cause activations to saturate immediately, and weights initialized too small cause activations to collapse toward zero. Both cases produce vanishing gradients before any learning occurs. Xavier initialization (Bishop and Bishop, 2024) scales the initial weights by a factor that depends on the number of inputs and outputs of each layer, keeping activations in a central non-saturating range. He initialization (He et al., 2016), designed specifically for ReLU activations, adjusts this scaling to account for the fact that ReLU zeros out half its inputs on average. Neither technique changes the training algorithm itself. Modern large model training uses variants of these approaches, and the initialization problem did not disappear with scale. It became more critical.
Batch normalization (Bishop and Bishop, 2024) keeps activations in a useful range throughout training rather than just at initialization. It normalizes the activations within each layer by subtracting their mean and dividing by their standard deviation across the current batch, meaning the set of training examples processed together in one training step. Sejnowski notes that this kind of normalization has a biological parallel (Sejnowski, 2018). The retina performs contrast normalization before passing signals to the visual cortex, adjusting its response based on the surrounding signal level rather than the absolute level. Batch normalization in artificial networks and retinal contrast normalization in biological vision solve structurally similar problems by structurally similar means. Transformers do not use batch normalization directly, since normalizing across a batch is problematic for variable-length sequences, and instead use layer normalization, which applies the same principle across the feature dimension rather than the batch dimension. The operation is the same. The axis of normalization shifted to fit the architecture.
Residual connections (He et al., 2016) create a direct path for gradients to travel backward. A residual block adds the input of a layer directly to its output before passing the result to the next layer. During the backward pass, the gradient can travel through this addition operation without passing through the layer’s weights at all. In a network with many residual blocks, gradients reach the earliest layers through a chain of addition operations, bypassing the multiplicative shrinking that causes vanishing. The original ResNet paper trained networks of 152 layers, a depth entirely out of reach before residual connections existed. Every transformer block uses the same structure, with both the attention sublayer and the feedforward sublayer adding their input directly to their output before the result moves to the next block. The residual connection that made the 152-layer ResNet trainable in 2016 is the same connection that made a 96-layer GPT-3 trainable in 2020, the last model for which OpenAI disclosed full architectural details.
What did NETtalk’s hidden layer learn?
NETtalk was a network trained by Sejnowski and Rosenberg in 1986 to convert written English text to phonemes, the basic units of spoken sound (Sejnowski, 2018). The input was a window of seven letters centered on the letter to be pronounced, and the output was a phoneme together with stress information. After training with backpropagation, the network achieved accuracy that, when passed through a speech synthesizer, was recognizable as English.
Get Eleventh Hour Enthusiast’s stories in your inbox
Join Medium for free to get updates from this writer.
The more significant result came from analyzing what the hidden layer had learned. Sejnowski applied cluster analysis, a method that groups data points according to similarity, to the activation vectors that the hidden layer produced for each input. An activation vector is a collection of numbers, one for each hidden unit, representing the network’s internal state for a given input. NETtalk had 16 hidden units, so each input produced a 16-dimensional activation vector. The clusters that emerged from this analysis matched the categorical structure of English phonology. Vowels grouped with vowels. Consonants grouped with consonants. Within the consonant cluster, further structure appeared that corresponded to distinctions linguists had identified through decades of analysis. The network had not been told about phonological categories. No label for vowel or consonant appeared in the training data. The categories emerged from the statistics of the mapping between written text and sound.
This result showed that networks trained with backpropagation do not simply memorize input-output pairs. They build internal representations that reflect the actual structure of the problem. The cluster analysis Sejnowski applied to NETtalk’s hidden layer is an early version of what researchers now call interpretability work. A common approach trains a probe classifier on transformer hidden states, a small linear model applied to frozen network weights to test what information a specific layer encodes. When a researcher asks whether a layer encodes grammatical number, or whether a direction in activation space corresponds to a semantic concept, they are asking the same question Sejnowski asked in 1986. What did the network learn on its own, without being told? That curiosity connects to a broader observation. Neural networks show an unexplained affinity for language (Sejnowski, 2018). In 2018 this read as a modest observation. A few years later it became the defining fact of the field.
Does the brain use backpropagation?
Backpropagation was inspired by biological neural systems, but it requires a mechanism that those systems do not appear to possess. The architecture it trains, layers of units connected by weighted synapses with nonlinear activation functions, was drawn from what neuroscience had established about cortical organization. The learning rule, however, does not match anything observed in biological tissue.
In the backward pass, each layer receives a signal encoding how the error at the output depends on that layer’s activations. This signal is computed from the weights of every subsequent layer and from the error at the final output. A weight in the first layer of a deep network must therefore receive gradient information that originated at the last layer, transmitted backward through every intermediate layer with the precision required to correctly attribute error to each weight. Neurons in biological systems do not receive signals of this kind, and no known feedback pathway carries information with the specificity that backpropagation requires.
Synaptic plasticity in biological systems is local, meaning a synapse changes its strength based on the activity of the two neurons it directly connects. This local rule is associated with Donald Hebb, who proposed in 1949 that neurons that fire together wire together, a principle that appeared in the Boltzmann machine’s weight update rule. Hebbian learning has no access to error signals computed at a remote output layer. Sejnowski acknowledges this tension (Sejnowski, 2018). The algorithm that made artificial neural networks powerful uses a learning rule that the biological brains which inspired it almost certainly do not use. The architecture drew from biology. The training algorithm did not.
What biologically plausible alternatives has the field proposed?
The biological implausibility of backpropagation motivated a search for learning algorithms that could achieve comparable results using only local information. Three directions have received sustained attention, and the most developed of them turns out to describe something that modern neural networks may already be doing implicitly.
Predictive coding frames learning as a process of prediction and error correction at every layer independently, without a global backward pass. Each layer maintains a prediction of the activity of the layer below it, and what travels upward through the hierarchy is not the raw signal but the prediction error, the difference between what the layer expected and what it received. If the prediction is accurate, nothing propagates upward. Only surprises move up the hierarchy, and learning at each layer is driven by its own local prediction errors with no need for gradient information from a distant output. Predictive coding traces back to Hermann von Helmholtz, who described visual perception in the nineteenth century as unconscious inference, meaning the brain generates top-down expectations and compares them against incoming sensory signals rather than passively receiving raw input. Sejnowski discusses this framework (Sejnowski, 2018), where the hierarchical structure of predictive coding maps directly onto the layered organization of the visual cortex. Karl Friston has developed predictive coding into what he calls the Free Energy Principle, which frames all of perception and action as minimization of prediction error (van Gerven and Bohte, 2017). Predictive coding and backpropagation have been shown to share structural similarities in how they update weights, raising the possibility that the brain and backpropagation-trained networks may be solving the same optimization problem through different mechanisms.
The connection to transformers makes predictive coding more than a neuroscience observation. When an attention head computes its output, the query vector encodes what the current position is looking for, acting as a prediction about what information is relevant. The key vectors encode what each position actually contains. The attention weights are the result of comparing that prediction against reality across every position in the sequence, with high weight indicating a match and low weight indicating a mismatch. The mechanism is not identical to predictive coding, but the structural parallel is real. The query is a top-down expectation, the attention weights reflect how well those expectations were met, and only the information that matches propagates strongly into the representation. Transformers may have arrived at something resembling predictive coding not through biological inspiration but through the pressure of solving sequence modeling efficiently.
The forward-forward algorithm (Hinton, 2022) replaces the backward pass with a second forward pass. The network runs once on real data, called the positive pass, and once on synthetically generated negative data constructed to represent incorrect examples. Each layer adjusts its weights to increase the representational quality of its activations for positive data and decrease it for negative data, using only local information at that layer, so the learning signal never travels backward through the network. Hinton noted explicitly that the forward-forward algorithm was inspired by the wake-sleep algorithm developed for the Boltzmann machine. The proposal closes a circle. The dissatisfaction with backpropagation’s biological implausibility that was present even in the Boltzmann machine era produced, four decades later, a proposed alternative that reaches back to that same period for its core idea.
Scellier and Bengio introduced equilibrium propagation (van Gerven and Bohte, 2017). The method connects learning to the energy minimization framework of Hopfield networks. A network is trained by finding two equilibrium states, one with the output fixed to the correct value and one running freely, and updating weights based on the difference between the activations in the two states. An equilibrium state is a stable configuration where the network is no longer changing. The update rule is local, and the learning signal comes from the network’s own dynamics rather than from an externally computed gradient. Both equilibrium propagation and the Hopfield network treat learning as a process of settling into low-energy configurations, the same energy-based view that underlies the Boltzmann machine. Whether any of these alternatives can match backpropagation’s empirical performance at scale remains an open question. The parallel between the query-key-value mechanism of attention and the prediction-error hierarchy of predictive coding suggests that the gap between biologically plausible learning and current practice may be smaller than it appeared when Rumelhart, Hinton, and Williams published their 1986 paper.
Conclusion
Backpropagation solved the problem of training multilayer networks and, in doing so, exposed two problems the field spent decades working around. The vanishing gradient problem was architectural. The sigmoid activation that made probabilistic reasoning tractable made deep training unstable, and the fixes required rethinking activations, initialization, normalization, and network structure. The biological plausibility problem was conceptual. The algorithm worked, but not in a way that illuminated how biological learning operates, the question that originally motivated neural network research. NETtalk sat between these two problems as a demonstration of what backpropagation reveals when it does work. The network discovered internal representations that reflect the structure of the problem domain without being told what that structure is.
References:
Rumelhart, D.E., Hinton, G.E., and Williams, R.J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536. https://doi.org/10.1038/323533a0
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. CVPR 2016. https://arxiv.org/abs/1512.03385
Hinton, G. (2022). The forward-forward algorithm: Some preliminary investigations. arXiv e-print, arXiv:2212.13345. https://doi.org/10.48550/arXiv.2212.13345
Bishop, C.M. and Bishop, H. (2024). Deep Learning: Foundations and Concepts. Springer. https://doi.org/10.1007/978-3-031-45468-4
van Gerven, M. and Bohte, S. (eds.) (2017). Artificial Neural Networks as Models of Neural Information Processing. Frontiers Media SA. https://www.frontiersin.org/research-topics/4817/artificial-neural-networks-as-models-of-neural-information-processing
Sejnowski, T.J. (2018). The Deep Learning Revolution. MIT Press. https://mitpress.mit.edu/9780262038034/the-deep-learning-revolution/
