Neural Networks that learn without Activation Functions

Hi to everyone reading this article,
My name is Sree Dhyuti and as a part of my Fall 2025 quarter at Northwestern, I took up the NEUROSCI_366 Brain Function Through the Lens of Computation course.

Here’s a cool paper I found as part of this course that talks about how Deep Neural Networks learn concepts just like humans do, with a mathematical explanation. Another interesting thing about this paper is that they use a neural network with no activation functions. Usually, when a neural network has no activation functions, we tend to assume that there won’t be any learning possible (since the network just becomes a simple linear transformation). But it was interesting to see how they argue that this isn’t the case and actually show non-linearity and therefore learning in neural networks without activation functions.

Link to the paper: https://arxiv.org/abs/1810.10531

(yes…it’s a paper from 7 years ago)

Motivation

This paper mainly talks about HOW neural networks (be it the brain networks or ANNs) learn knowledge abstraction.

Eg: You know what a “dog” is. You’ve probably seen hundreds of different dogs: chihuahuas, golden retrievers, etc.

(src) A collections of different dogs.

All these dogs have different properties in terms of their looks, behaviour, etc. But after seeing a lot of dogs in your life, somehow your brain extracted an abstract concept of “dog”. You can now classify an animal you have never seen before into “Dog” or “Not Dog” based on this abstract concept that is created in your brain over time. How did that happen?

This paper dives deeper into finding explanation for this phenomenon of the brain by drawing clear parallels from the concepts of Neural Networks.

Introduction

As a human, you collect a lot of experiences (or collect data), think (or process the data), and acquire knowledge while doing so. This mental process of acquiring knowledge is called cognition.

When you were a child, you didn’t sit down with a textbook to learn what “dog” means. You saw individual dogs, and your brain, through some mysterious process, figured out the pattern to check for, to identify a dog.

Over time, as you grew up, you’ve seen more and more data, leading to more and more learning.

And usually all the data our brain processes is usually unsupervised data in which we identify patterns from and label them over time in our heads.

An observation on this continual learning process:
Broader categorical distinctions are generally learned before finer-grained distinctions.
For example, a human baby will learn to differentiate between “dogs” and “fish” before they learn to differentiate between a “chihuahua” and a “dalmatian.” This is because dogs and fish are far more different from each other than a chihuahua and a dalmatian are.
=> Humans tend to learn coarse/broad categories first and over time tune their models (brains) to learn the finer differences.

Why does this coarse-to-fine learning happen?
This paper provides a mathematical explanation for this phenomenon using a simple deep linear neural network. Even though the model is linear (no activations functions like in a normal Neural Network), the learning dynamics turn out to be nonlinear and rich enough to reproduce many cognitive behaviors found in children.

The authors show that when a neural network/brain is exposed to many examples (Eg: many animals and their properties), it slowly organizes its internal representation of the world.

But importantly:
It learns large, broad distinctions first
(Eg: living vs. non-living, animals vs. plants)

Then progressively learns finer distinctions
(Eg: birds vs. fish → canary vs. robin)

This is not because the network is explicitly told about these categories.
It emerges naturally from statistics of the environment and the way gradient descent updates weights.

Deep Linear Neural Network

To understand this framework let’s take an example:

Let’s say we have some items and we know some properties of them.

Items: things like canary, salmon, oak, rose,….
Properties / Features: properties like can fly, can swim, has bark, has petals, can move,….

I used GPT to populate a Item-Properties matrix as shown below:

This has 3 levels.

Level 1 : Universal properties that apply to every item
Level 2 : Animals vs Plants, properties that can help us differentiate between animals and plans
Level 3 : Properties that can help in differentiating between individual item

If the network encounters an item, such as a canary, the neural circuits of the brain produce a signal that can be considered mathematically as follows:

Over time, the network experiences several such {x, y} pairs. The aim of the Deep Linear Network is to use these observed {x, y} pairs and predict an item’s properties y from its perceptual representation x.

Src: the original paper

When I first read this, my question was:
How would a linear network learn if we build a neural network like this?

Given that there is no activation function, there is no non-linearity. If we have two layers with weights W1​ and W2​, the output is

y = W_2 (W_1 x)

With no activation between the layers, this could just collapse to

y = (W_2 W_1) x = W x

which is just a single linear transformation. So the whole point of having multiple layers is gone right?
Atleast, this logic makes sense right?
This is how I thought too. But the authors of the paper say that even though the network is linear, the learning dynamics is still non-linear and hence accommodates learning.

The image below is taken from the paper and shows that over time, a deep Linear neural network is also able to segregate between different items in the data.

which is just a single linear transformation. So the whole point of having multiple layers is gone, right?

I chumma (for fun) built a neural network without any activation functions.

class DeepLinearNetwork(nn.Module):
    """Deep Linear Network - NO activation functions"""
    
    def __init__(self, input_dim, hidden_dim, output_dim, init_scale=0.01):
        super().__init__()
        
        # Two linear layers 
        self.W1 = nn.Linear(input_dim, hidden_dim, bias=False)
        self.W2 = nn.Linear(hidden_dim, output_dim, bias=False)
        
        nn.init.normal_(self.W1.weight, mean=0, std=init_scale)
        nn.init.normal_(self.W2.weight, mean=0, std=init_scale)
        
    def forward(self, x):
        """Forward pass - all linear"""
        h = self.W1(x)     
        y_hat = self.W2(h) 
        return y_hat, h
    
    def get_composite(self):
        """Get composite weight W²W¹"""
        W1 = self.W1.weight.detach().cpu().numpy()
        W2 = self.W2.weight.detach().cpu().numpy()
        return W2 @ W1

# Create network with expanded dimensions
N1 = 12  # Input dim (12 items - 6 animals, 6 plants)
N2 = 8   # Hidden dim (bottleneck - forces abstraction)
N3 = 27  # Output dim (27 properties - hierarchical)

model = DeepLinearNetwork(N1, N2, N3, init_scale=0.01)

Upon training this (will add a link to the code in the end, it’s too simple regardless), I was able to see that the model was learning indeed.

Learning over time

Notice that in epoch 1, all the data points are scattered randomly without any recognizable structure. By epoch 780, the trees (green) and flowers (pink) begin clustering together, and the other categories also start forming broad groups. By epoch 1980, even the fine-grained categories have separated cleanly.

What you should see:
- Early epochs: All items are mixed together with no clear separation.
- Middle epochs: Broad categories emerge, such as animals vs. plants.
- Late epochs: Fine-grained distinctions appear like birds, fish, mammals, trees, flowers, vegetables.

Get Sree Dhyuti Nimmagadda’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

This is exactly how children learn:
They first identify coarse categories (animals vs. plants), then make medium-level distinctions (birds vs. fish vs. mammals), and finally learn fine-grained categories (canary vs. robin).

Acquiring Knowledge

These deep linear neural networks learn their weight/synaptic weights through an incremental error-corrective process. This is analogous to online SGD. Each time it sees an {x, y} pair, it:

1. Makes a prediction: ŷ = W² W¹ x
2. Computes error: How different is ŷ from y? (We used MSE)
3. Updates weights to reduce error

optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
criterion = nn.MSELoss()

for epoch in range(2000):
    y_pred, h = model(X)
    loss = criterion(y_pred, Y)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

In a 2-layer linear network, the outputs from each layer are:

(typed on latex and screenshotted cause medium doesn’t support formulae :))

For each training example i, we have {xi, yi} where yi is the target output. If we are using a MSE loss function, then we have loss:

The below are standard gradient descent updates for a linear 2-layer network for a single training example:

To do this over a set of training datapoints, we basically need to do an average of the above over the datapoints. So instead of looking at a single data point, we now look at their expected values under this data distribution:

To express these expectations compactly, we define the following correlation matrices of the environment:

Intuitively,

• the sigma_xy correlation terms tells us which input patterns tend to co-occur with which output patterns
• sigma_x tells us how the inputs are structured with respect to each other (which input directions have high variance and how they co-vary)

Gradient descent can be approximated by a continuous-time differential equation and we write:

T -Time constant

Using the definitions we have of signma_xy and sigma_x, these gradient descent updates can be written as:

src: the original paper

Now look closely at the term:

sigma_yx is the true input–output correlation in the environment.
W2 W1 sigma_x is the input–output correlation predicted by the network with its current weights.

So this difference is the error in the correlation space. It measures how far the network’s internal mapping is from the real statistical structure of the data.

As per the weight update formulae,

• W1 changes in the direction of W2 times this correlation error. This is backpropagation: the correlation error at the output is sent back through W2 to adjust W1
• W2 changes in the direction of this correlation error times W 1T , i.e, using the current hidden representation as context

In other words,

Learning is driven by the difference between how inputs and outputs are related in the environment and how the network currently represents that relationship

These equations reveal that learning dynamics in even in our simple linear network can be highly complex: the second order statistics of inputs and outputs drives weight changes through coupled nonlinear differential equations with up to cubic interactions in the weights.

I hope it was clear so far. Feel free to email me at sreedhyutin@gmail.com if you didn’t understand something.

Next is a math derivation to prove that Neural Networks learn in a hierarchical fashion during training, i.e, they learn plants vs animals in the first few epochs and then learn lily vs daisy in the next few epochs and so on..

SVD of the input–output structure (the “semantic modes”)

U contains orthonormal directions in output space,

V contains orthonormal directions in input space,

S is diagonal with singular values s1 ≥ s2 ≥ · · · ≥ 0.

Each triplet (Ui, Si, Vi) represents a mode. Each of these triplets represents information in a specific direction.

Mode (U1, S1, V1) represents a input pattern V1 that’s associated with U1 with a strength S1.

Modes with large S values represent broad categories like plans, animals, etc.

And, modes with smaller S values are fine-grained categories like daisy flowers, lily flowers, cats, dogs, etc.

For a specific mode ‘k’, the strength of the network’s representation of that mode, denoted ak(t), obeys approximately:

ak = sk in the best case

On solving this equation (by integrating wrt time), we get:

During training, ak(t) gets updated and gets as close to sk as possible.

Also note that ak(t) is directly proportional to sk and inversely to time t

This means that modes with large Sk get learned in less time. The network will first learn the mode with the largest S value and then the 2nd largest S value and so on..

After training my model, I calculated the custom speed of learning for each modes as (1/epochs). The results are shown below:

Scaled the speeds by multiplying with 10000 for better visibility

Zoom and observe that the model learns in perfect hierarchy. You should try it with may be a CIFAR dataset and see the same results.

I really hope I was clear at this point :)

The paper also discussed other topics like

• explanations to why Neural networks hallucinate in mid-training phases
• a formula for trying to identify if a model can learn a specific category or not (search for category coherence term in the paper)
• Inductive Projections….

Feel free to read the paper for better understanding.

My biggest takeaways from the paper:

• Deep networks and children learn semantic knowledge in similar ways , not because of built-in rules, but because of the statistical structure of the world and the dynamics of gradient-based learning
• Neural Networks without activation functions can learn too
• Learning happens in a hierarchy-fashion