The Deep Learning Revolution
Book Review
Introduction
Terrence J. Sejnowski spent four decades working at the intersection of neuroscience and neural networks, long before the field was called deep learning. He co-invented the Boltzmann machine with Geoffrey Hinton, helped establish the Neural Information Processing Systems conference in 1987, trained NETtalk, and took part in early debates and developments around backpropagation, convolutional networks, and reinforcement learning. The account in The Deep Learning Revolution (2018) reflects this position. It presents not an external view of a technology, but a record of ideas developed over many years under sustained resistance.
Two caveats are important before reading. The book was published in 2018, and the application chapters reflect the expectations of that period. They should be read with that context in mind. One sentence captures the gap between that moment and what followed. “We cannot exclude the possibility that some very large generative network will someday start talking, and we can ask it for explanations.” That possibility became real a few years later. The foundational chapters do not have this limitation. They describe ideas that remain central to the field.
This review organizes the book around a set of core questions. These questions focus on the biological, algorithmic, and institutional foundations of deep learning.
Why did symbolic AI fail where neural networks succeeded?
Symbolic AI treats intelligence as a process of applying rules to symbols. Neural networks learn from data by adjusting internal weights. This difference helps explain why symbolic AI ran into limits on the hardest problems.
Symbolic AI did not fail because of a lack of intelligence or effort. Researchers such as Marvin Minsky, Allen Newell, and Herbert Simon were serious scientists working on genuinely difficult problems. The limitation was structural. The approach rested on assumptions about intelligence that did not hold for the most challenging tasks.
Sejnowski identifies four properties of biological intelligence that symbolic AI could not account for. The first is that intelligence depends more on pattern recognition than on rule application. Many everyday tasks such as recognizing a face, understanding a sentence, or navigating a familiar space happen without conscious rule following. Systems like MYCIN, an early medical expert system for diagnosing infections and recommending antibiotics, required experts to write down their knowledge as rules, but skilled practitioners rely on patterns built through years of experience that are hard to express.
The second is that learning comes from doing rather than from being programmed. A system that only knows what it has been told is limited by what can be described in advance. In symbolic systems, this meant encoding knowledge as rules written by experts, covering only the cases they could anticipate. That covers only a small part of what a skilled person actually knows. Much of that knowledge comes from repeated exposure to many examples, where patterns are learned through experience rather than stated explicitly. This allows practitioners to handle new situations they have not seen before.
The third is that the brain does not operate through explicit logic. In one classic experiment, a rule is given, if a card has an even number on one side, the other side must be red, and one side of a set of cards is shown. The goal is to choose which cards must be checked to test the rule. Only a small fraction of people get this right when the problem is presented in abstract form. When the same structure is presented in a familiar social setting, for example checking cards that show a drink on one side and an age on the other where anyone drinking alcohol must be over 18, most people answer correctly. This is known as the Wason selection task. The logic is identical in both cases. Human reasoning developed to handle social situations, detect violations, and navigate the physical world. It was not built for working through abstract symbols.
The fourth is structural. The brain is made up of billions of neurons working in parallel. Each is slow and unreliable, yet together they produce robust intelligence through collective activity. This is very different from a system that processes one instruction at a time.
These properties were beyond what symbolic AI was designed to handle, and they point toward a different computational approach. Knowledge is distributed across parameters rather than written as rules. Learning occurs through exposure to examples. The architecture mirrors the parallel structure of the biological system that inspired it, with many simple units working together rather than one unit following instructions in sequence. This is the approach that Sejnowski and his collaborators developed over several decades.
How did biological vision research shape the architecture of deep learning?
The connection between biological vision research and convolutional neural networks is precise and grounded in experimental results. The relationship can be traced step by step.
The story begins in the retina. Stephen Kuffler showed in 1953 that retinal ganglion cells do not simply measure brightness. They measure contrast, the difference between the brightness of a small central region and the surrounding area. One type of cell fires when the center is bright and the surround is dark. Another type fires when the center is dark and the surround is bright. The retina does not send a raw brightness map to the brain. From the first stage of processing, it sends a contrast-enhanced signal that highlights edges and boundaries. This occurs before any information reaches the visual cortex.
David Hubel and Torsten Wiesel extended this work into the cortex in 1962. Neurons in the primary visual cortex, called simple cells, respond to oriented edges rather than spots of light. Each simple cell has a preferred angle and fires when a bar of light appears at that angle within its receptive field. A second type, called complex cells, responds to the same orientations but does not require the edge to appear at a fixed position. A simple cell fires only when the edge appears at a specific location within its receptive field, while a complex cell continues to fire as the edge moves within that region. This allows the same feature to be recognized even when its position shifts slightly.
These steps map directly onto the structure of a convolutional neural network. A convolutional layer applies a set of small filters across an image. Each filter responds to a specific local pattern such as an edge or a particular orientation. The same filter scans every location, producing a map that shows where that pattern appears. A pooling layer follows each convolutional layer. It summarizes the responses within small regions. Convolutional layers detect features at specific locations, similar to simple cells, while pooling layers make those detections less sensitive to position, similar to the behavior of complex cells. Stacking these layers builds a hierarchy. Early layers detect edges and orientations. Middle layers combine these into corners and textures. Deeper layers represent shapes and objects. Yann LeCun demonstrated in the late 1980s that networks with this structure could learn effectively on real tasks at scale. The design matched what Hubel and Wiesel had measured more than two decades earlier.
A third biological insight connects to modern training methods. In 1961, Horace Barlow proposed that the visual system does not simply pass along everything it receives. It removes redundancy from its representation of natural scenes. Nearby pixels in a natural image tend to look alike, and transmitting that repeated information is wasteful. The visual system, Barlow argued, finds a more compact representation that keeps what is informative and discards what is not. Shannon’s source coding theorem from 1948 provides the mathematical foundation for this idea. Shannon showed that the most efficient way to represent any signal is to reduce the parts that repeat or are predictable. Applied to vision, this means that a good visual representation should capture edges, contrasts, and changes rather than raw brightness values, because those are the parts that carry information. Barlow’s conjecture sat largely as an intuition for decades. But the principle it describes shows up directly in two techniques that are now standard in deep learning, batch normalization and data augmentation with lighting variation.
Batch normalization adjusts the activations within each layer during training. It removes the absolute scale and keeps the relative differences, similar to how the retina drops absolute brightness and keeps contrast. The network learns to respond to patterns rather than to fixed values.
Data augmentation with random lighting changes follows the same logic. During training, the brightness and contrast of images are varied randomly. A model trained this way cannot rely on fixed brightness values. It learns to recognize shape from the pattern of shading instead, which is how the visual system handles changes in illumination.
The Ramachandran shading illusion shows why this matters. A grid of shaded circles appears as raised shapes when the bright region is at the top and as hollow shapes when the image is inverted. The visual system assumes that light comes from above, a prior shaped by experience and evolution. The same image produces two different perceptions depending on orientation. The reversal occurs immediately and without effort.
What does it mean to combine fields that were never meant to combine?
Sejnowski’s career follows a consistent pattern. He moved from physics to neuroscience, brought signal processing into neural network research, and continued to draw methods from one field into another. This boundary crossing is not a side feature of his work. It is the source of many of his most important contributions.
The cocktail party problem is a clear example. The task is to separate individual signals from a mixture, such as isolating one voice in a room full of overlapping conversations. The signal processing community had developed mathematical tools for separating signals based on statistical properties, but these methods required some prior knowledge of the sources. The neural network community had explored learning-based approaches that could adapt from data, but lacked the statistical framework needed to guarantee separation of independent sources. Sejnowski and Tony Bell combined these approaches. They used the information maximization principle from Shannon’s information theory to derive a learning rule that separates statistically independent sources from a mixture without prior knowledge of the signals. The result was Independent Component Analysis (ICA), published in 1995. Neither field alone had reached this solution. The combination made it possible.
What makes ICA particularly notable is what followed. The method was developed by thinking about how the brain separates sounds. It was then applied to study the brain itself. EEG recordings capture mixtures of electrical activity from many neural sources, combined by the effects of the skull and surrounding tissue. Traditional EEG required subjects to remain still because movement introduced artifacts that could not be separated from neural signals. ICA changed this. It separates brain activity from eye blinks, head movement, and heartbeat signals mathematically. Subjects could move, walk through virtual environments, and interact with others while brain activity was recorded. A method inspired by biology became a tool for studying biology. This two-way exchange between neuroscience and AI runs throughout the book.
David Marr represents a different thread in this story. He was one of the most systematic thinkers on biological vision, and he did not live to see his framework confirmed. Marr died of leukemia in 1980 at age 35. His book Vision was published after his death in 1982. He argued that any information-processing system should be understood at three levels. The computational level asks what problem is being solved. The algorithmic level asks what procedure solves it. The implementational level asks how it is realized physically. This framework became foundational in cognitive science and computational neuroscience. Deep learning followed a different path. It discovered algorithms that worked before there was a clear understanding of the problems they solved. The gap between Marr’s top-down framework and the bottom-up empirical approach of deep learning remains unresolved. The field benefits from both.
Misha Mahowald belongs to this same thread. As a graduate student of Carver Mead at Caltech in 1982, she built the first silicon retina. The center-surround responses that Kuffler had measured in living retinal cells in 1953 became the operating principle of a chip three decades later. Her design translated this contrast detection mechanism directly into analog hardware, detecting changes in light intensity continuously without capturing frames, the way biological retinas operate. This was the first demonstration that biological sensory processing could be implemented in hardware rather than simulated in software.
That distinction defines neuromorphic engineering as a field. Where conventional computing simulates biological processes on general-purpose hardware. Neuromorphic engineering builds circuits whose physical operation mirrors the mechanisms of biological neurons. She was inducted into the Women in Technology International Hall of Fame in 1996 for this work. She died the same year at age 33. The principle she implemented remains central to neuromorphic engineering today. The book is dedicated in part to her memory.
Get Eleventh Hour Enthusiast’s stories in your inbox
Join Medium for free to get updates from this writer.
How did the neural network community survive twenty years of institutional rejection?
The AI winters were not caused by bad science. The core ideas behind neural networks were largely sound. The problem was a mismatch between what the hardware of the time could support and what the approach required. Scale, data, and parallelism were necessary, and none were available for decades. The community had to maintain confidence in ideas that could not yet be fully demonstrated while working in a field that had lost funding and faced active resistance.
In 1988, Sejnowski served on a committee tasked with developing cognitive neuroscience as a field. Jerry Fodor, a philosopher who argued that the mind operates as a symbol-processing system, opened the meeting by declaring that cognitive neuroscience was not a science and never would be. When John Bruer, president of the McDonnell Foundation, pointed out that Fodor had confused his foundation with a hamburger chain while criticizing it, Fodor continued without pause. When Mortimer Mishkin, a neuroscientist studying vision and memory, asked Fodor to describe discoveries from his own lab, Fodor offered a vague response about an event-related potential experiment. At that moment a fire drill was called. Outside, Mishkin remarked, “Those are pretty small potatoes.” When the drill ended, Fodor had left. A philosopher who arrived certain enough to dismiss an entire field left without defending his own work.
At the 1983 Cognitive Science Society symposium, Zenon Pylyshyn, a cognitive scientist and strong advocate of symbolic approaches, objected to the work Sejnowski and Hinton were developing and declared that it was not computation at all. During the exchange, he poured a glass of water on the stage. Others dismissed the approach as mere statistics. These were established researchers defending a dominant paradigm against connectionism, the view that intelligence emerges from networks of simple units that learn from data rather than from explicit symbolic rules. The intensity of the reaction reflected the scale of the threat.
The response from the neural network community was not debate but demonstration. In their 1969 book Perceptrons, Minsky and Papert had argued that neural networks were fundamentally limited because single-layer networks could not solve problems where no linear separator exists. One of the key tools that answered this challenge was the Boltzmann machine, published by Hinton, Sejnowski, and Ackley in 1985. Unlike single-layer networks, the Boltzmann machine used multiple layers. The additional layers allowed the network to develop internal representations of structure that was not visible in the raw input. The model also used a probabilistic learning procedure inspired by physical systems, allowing it to explore possible configurations rather than follow a fixed path. This gave it capabilities that single-layer networks did not have.
In 1986, Sejnowski, Kienker, and Hinton tested this directly using symmetry detection, classifying whether a pattern has a vertical, horizontal, or diagonal axis of symmetry, the same problem Minsky and Papert had used to make their case. A Boltzmann machine with 100 input units and 16 hidden units solved the task. The hidden units developed internal representations that captured the symmetry structure without being told how. The answer to the 1969 argument was a working system.
In 2006, Dartmouth College hosted AI@50, marking fifty years since the original conference where McCarthy, Minsky, Shannon, and others launched artificial intelligence as a field. Several of the original participants returned for the occasion. In his closing talk, Minsky expressed disappointment with the direction AI had taken. His view was that the field had turned toward applications instead of general intelligence.
Sejnowski presented work on reinforcement learning and TD-Gammon, a program that had taught itself to play backgammon at world championship level through self-play, discovering strategies that human experts had not found. The learning mechanism it used matched how the brain is thought to learn from reward. Dopamine neurons fire when an outcome is better than expected and remain quiet when it matches expectations, producing a signal that adjusts behavior over time. TD-Gammon used the same mathematical structure.
Minsky was not impressed. He dismissed it as a mere game. A system with these properties was reduced to an example of limited scope.
During the question period, Sejnowski stood and asked Minsky directly, “There is a belief in the neural network community that you are the devil responsible for the neural network winter in the 1970s. Are you the devil?” Minsky began a response about mathematical limitations. Sejnowski interrupted and asked for a yes or no answer. Minsky paused, then said yes. The exchange is notable for its directness. Minsky acknowledged the effect of his 1969 book at the same institution where the field had begun fifty years earlier. One year later, Hinton’s work on deep belief networks marked the beginning of the modern deep learning revival. The path to broader capability ran through the domain-specific learning Minsky had dismissed.
The account becomes more complex when considering what Sejnowski reveals about Minsky’s early work. His 1954 doctoral dissertation at Princeton was a theoretical and experimental study of neural networks. He built small networks from electronic components to study their behavior. The Mathematics Department did not have the expertise to evaluate the work and sent it to the Institute for Advanced Study. The response was that if this was not mathematics at the time, it would be in the future. That was sufficient for the degree. This early work reflects a different position. The same figure who later closed the door on connectionism began as one of its pioneers.
What did energy-based networks contribute that feedforward networks could not?
Most neural networks before the 1980s were feedforward networks, built to transform inputs into outputs. Given an image, produce a label. Given a letter, produce a sound. The network learned a mapping and nothing more. It had no internal model of what the data looked like, no ability to complete a partial pattern, and no way to generate new examples from what it had learned.
John Hopfield changed that in 1982 by borrowing an idea from physics. He showed that a network of interconnected units could be described using the same mathematics that physicists use for systems that settle into stable states. Memories in such a network are not stored at specific addresses the way a computer stores data. They are stable configurations that the network moves toward. Present a partial or degraded version of a stored pattern and the network fills it in, the way a person recognizes a song from the first few notes. This is associative memory, and it showed that neural networks could retrieve and complete patterns rather than only map inputs to outputs.
The Boltzmann machine extended this energy-based approach. Where the Hopfield network could only retrieve patterns it had been given, the Boltzmann machine could learn the structure of data. It introduced a third layer called hidden units, sitting between the input and output layers, with no direct connection to the outside world. Their values are determined entirely by the activity of the units around them, allowing the model to form internal representations that explain what is observed. It replaced the deterministic update process with a probabilistic one, controlled by a temperature parameter, allowing the network to explore different configurations before converging. The learning procedure involved two phases that Sejnowski named wake and sleep, echoing how the brain is thought to consolidate experience during sleep. The wake phase moves the network toward the data it observes, while the sleep phase reduces patterns it generates on its own.
The lineage from these ideas is direct. The restricted Boltzmann machine simplified training. Hinton’s deep belief network in 2006 stacked these models into deeper architectures and helped revive the field after a long period of limited progress. The variational autoencoder, the GAN, and the diffusion model that underlies many current image generation systems all follow the same generative approach. The core question is how a system can learn the structure of data well enough to produce new examples. Earlier feedforward networks learned mappings from inputs to outputs. Energy-based models learned the structure of the data itself. That difference is what this line of work introduced and what the field continues to build on.
How did a backgammon program foreshadow modern reinforcement learning?
Backgammon is a board game where players move pieces based on dice rolls. Unlike chess or Go, the dice make every position the result of both deliberate choice and irreducible randomness. A program cannot plan many moves ahead with certainty because future dice rolls are unknown. Expert human players do not calculate deep move trees. They judge positions. Given a board, across the range of likely dice outcomes over the next several moves, how favorable is the situation? This is probabilistic judgment under uncertainty, and it is what TD-Gammon learned to do.
Gerald Tesauro applied Richard Sutton’s temporal difference learning algorithm to backgammon in 1992, building on earlier work with Sejnowski. Temporal difference learning is a method where a system learns by comparing its predictions about future outcomes with what actually happens, adjusting its estimates continuously rather than waiting for a final result. The core challenge is the temporal credit assignment problem. In a game that lasts many moves, the only definitive signal is win or lose at the end. Which moves were responsible? Temporal difference learning addresses this by making predictions at every step. At each board position, the network estimates the probability of winning from that state. After a move, the network evaluates the new position and produces a revised estimate. The earlier estimate is then adjusted toward this new estimate, along with any immediate reward. In this way, corrections propagate through the sequence without waiting for the final outcome. Repeating this process over millions of self-played games allows the network to steadily improve its judgment of each position.
No human expert encoded strategy and no labeled positions were provided. TD-Gammon started from random play and reached world championship level. After 1.5 million self-played games, world champion Bill Robertie played it to a draw. Kit Woolsey, a leading expert, concluded that its positional judgment exceeded that of any human he had seen. TD-Gammon also discovered strategies that human experts had not found. The sequence of concepts it learned, first bearing off pieces, then hitting opponent pieces, then avoiding being hit, and then building points, mirrors the progression human players develop through experience. It arrived at the same strategic understanding faster and went further.
The connection to neuroscience is direct. Wolfram Schultz showed that dopamine neurons signal reward prediction errors. They fire when an outcome is better than expected, remain quiet when it matches expectations, and decrease activity when an expected reward does not arrive. This pattern matches the temporal difference error signal. An algorithm developed to model learning in animals turned out to describe what dopamine neurons do in the brain.
The same principle runs through modern systems. It connects animal conditioning experiments, Sutton’s algorithm, TD-Gammon, AlphaGo and AlphaZero, and reinforcement learning from human feedback in language models. When a human evaluator compares two model outputs and selects the better one, that signal is used to adjust the model’s behavior in a way that mirrors temporal difference updates. Process reward models extend this idea by assigning rewards at intermediate steps rather than only at the final output. The algorithm that enabled TD-Gammon in 1992 now appears in the training of the most capable language models.
How does the book read in 2026?
Reading a 2018 book about AI in 2026 produces a specific kind of double vision. The foundational chapters read as current and relevant. The application chapters reflect what the moment felt like before several predictions met reality. Progress on autonomous trucking has fallen well short of the 2028 timeline the book anticipated. The gap between current capability and deployment-ready reliability remains significant, and many well-funded autonomous trucking startups have shut down or scaled back. The idea that whoever has the most data wins underestimated the role of algorithmic innovation and access to compute. It also could not anticipate the semiconductor export controls that became a central instrument of US-China AI competition after 2022.
Sejnowski correctly identified that general-purpose computers would no longer be practical for the most compute-intensive applications and that special-purpose chips with different architectures would dominate, using mobile phones as an early example. That direction was correct. What followed was faster and more consequential than the book suggested. The special-purpose chip became the Nvidia GPU, and the GPU became the most strategically important semiconductor, shaping export controls, industrial policy, and trillion-dollar valuations. Sejnowski pointed toward neuromorphic computing as the architecture that would fill this role. That prescription did not hold. The GPU delivered massively parallel digital compute that reduced the immediate need for a fundamentally different approach. Neuromorphic computing remains a research direction. The GPU prevailed.
Sejnowski also observed that the commercialization of technology developed through basic research often takes about fifty years, citing quantum mechanics leading to computers and DNA research leading to gene therapy. Deep learning appears to have moved faster, from foundational work in the 1980s and 1990s to large-scale commercial impact in the 2010s and 2020s. This acceleration was driven by the convergence of GPU compute, internet-scale data, and the transformer architecture. One observation from Chapter 15 reads as description rather than forecast in 2026. Which companies and countries will control this technology depends on the investments and bets being made today. The bets made between 2018 and 2022 determined who leads today.
Several observations from the book carry different weight in 2026. In the limitations section of Chapter 8, Sejnowski asks whether networks could be trained to give explanations by making explanations part of their training data, framed as an open question. In Chapter 9, he notes that neural networks show an affinity for language that was not yet understood. In Chapter 12, he uses next word prediction as a simple illustrative example. All three were careful observations in 2018. The first describes instruction tuning and chain-of-thought training. The second became central to the following decade. The third became the training objective of the most widely used AI systems.
One passage requires different attention. Sejnowski describes the possibility of training a generative network on recordings of a deceased actor to produce new performances, presented as a neutral observation about future capability. That possibility has since been realized. Legal disputes over digital likeness, films that recreate actors who died during production, and music generated in the voice of living artists without consent are now part of the industry’s reality. The technical direction was accurate. The ethical consequences arrived with the capability in ways that a 2018 observation could not fully anticipate.
The work of Hopfield and Hinton, described throughout the book, was recognized with the 2024 Nobel Prize in Physics. Work presented as historical background in 2018 was later recognized as a fundamental contribution to understanding how physical systems store and process information. The gap between the water-pouring at the 1983 symposium and the Nobel Prize in Stockholm is forty-one years. That arc is what the book ultimately captures.
Conclusion
The Deep Learning Revolution is a book about foundations, biological, algorithmic, and institutional, told by someone who helped build them. It traces how ideas moved from neuroscience into machine learning and back again. The hierarchy of the visual cortex shaped convolutional networks. Energy minimization led to the Hopfield network and the Boltzmann machine. The temporal difference algorithm became central to modern reinforcement learning. The ICA framework connects neuroscience and AI in both directions. These chapters remain as relevant in 2026 as they were in 2018.
Sejnowski organizes this history around a single question. How did a small community keep an unfashionable set of ideas alive for two decades despite institutional resistance, limited funding, and strong opposition from dominant approaches? The answer is empirical persistence. The ideas continued to work even when support was limited, and over time the necessary data and computational power became available.
Reference:
Sejnowski, T. (2018). The Deep Learning Revolution. MIT Press. https://mitpress.mit.edu/9780262038034/the-deep-learning-revolution/
