Superposition in Toy Models

The researchers struggled to answer a simple question for years.

How does a neural network store knowledge when it has fewer neurons than the number of concepts it appears to understand?

The obvious answer is to give every concept its own neuron. If a model understands programming, perhaps one neuron represents loops, another variables, another functions, and another recursion. If it understands language, maybe separate neurons represent nouns, verbs, punctuation, and sentiment.

It is an interesting way as it resembles how humans organize information. But, it is almost certainly wrong.

As researchers began working on modern neural networks, they saw a recurring pattern. Individual neurons rarely behave like neatly labeled storage boxes. Instead, they behaved like multitaskers. A neuron that is activated for legal terminology might also respond to mathematical notation. Another might react to computer code, bullet lists, and quotation marks. The more models researchers inspected, the harder it became to argue that every neuron represented exactly one idea.

Either neural networks were learning messy, disorganized representations, or researchers were asking the wrong question.

The second possibility turned out to be far more interesting.

• Perhaps neurons were never meant to represent concepts individually.
• Perhaps concepts themselves were…