Independent Component Analysis
How a signal processing algorithm became a lens for understanding neural networks
Introduction
The cocktail party problem captures a challenge that recurs across signal processing and machine learning. Several microphones record a room where multiple people speak at the same time. Each microphone captures a mixture of all the voices. The task is to recover each individual voice from those mixtures, with no prior knowledge of what any voice sounds like, how many speakers are present, or how the room is arranged. Recovering sources from mixtures under these conditions is called blind source separation, where “blind” refers to operating without labels, templates, or knowledge of how the signals were combined.
The infomax algorithm, introduced by Anthony Bell and Terrence Sejnowski in 1995, addresses this problem through a neural network trained to maximize mutual information. Mutual information is a quantity from information theory that measures how much knowing one variable reduces uncertainty about another. When the network maximizes it across all outputs, those outputs become statistically independent from each other, which is the condition that holds when each output corresponds to a single source.
The reach of infomax extended beyond audio separation. Bell and Sejnowski drew simultaneously on information theory, signal processing, and computational neuroscience at a time when researchers typically worked within a single discipline. Sejnowski later traced this connection in The Deep Learning Revolution (Sejnowski, 2018), linking blind source separation to the way the visual cortex encodes images. The same decomposition principle reappears in the attention mechanisms of modern language models, in the theoretical foundations of generative models that learn disentangled representations, and in the internal structure that trained networks develop without being directed to.
This post traces Independent Component Analysis (ICA) through that arc, from the original 1995 algorithm to the problems and structures that define modern machine learning.
Why did it take nearly five decades to connect neural coding to information theory?
In 1961, Horace Barlow put forward a conjecture that the early visual system reduces statistical redundancy in the retinal input, meaning it removes repetitive or predictable patterns from the visual signal before passing it deeper into the brain. Barlow argued that simple cells in the primary visual cortex, the neurons that fire in response to oriented edges, encode the visual scene efficiently by carrying as much information as possible while keeping neural activity low. The conjecture drew directly on the information theory Shannon developed in 1948, but confirming it required a mathematical tool that did not yet exist.
Principal Component Analysis (PCA) was the standard tool for learning compact representations through most of the following decades, and understanding its limitation is what makes Barlow’s problem concrete. PCA finds directions in data that are uncorrelated. Uncorrelated means the covariance between any two components is zero, where covariance measures whether two variables tend to move together. When covariance is zero, the variables have no linear relationship. Independence is a stronger condition than uncorrelatedness. Two variables can be uncorrelated and still depend on each other through higher-order relationships, meaning patterns that involve more than pairwise linear associations. As an example, a variable x drawn uniformly from [-1, 1] and a variable y = x² have zero covariance, but knowing x determines y completely. PCA would treat them as independent when they are not.
Barlow’s conjecture required independence in the full statistical sense, and PCA could not provide it. Natural images carry higher-order structure that second-order methods, those that only examine covariance, cannot detect. The correlations between oriented edges, the non-Gaussian statistics of natural textures, and the sparse distributions of filter responses are all invisible to PCA. Non-Gaussian means the data does not follow a bell-curve distribution, which matters because covariance-based methods are designed around that assumption. Confirming Barlow’s conjecture required a method that reached beyond covariance into higher-order statistics.
Bell and Sejnowski provided that method in 1995, and its result connected the algorithm directly to biology. Infomax maximizes the entropy of the output of a nonlinear function applied to the data, where entropy measures the amount of information or unpredictability in a signal. The nonlinearity is what makes the objective sensitive to higher-order statistics rather than covariance alone. When trained on natural image patches, the network learned filters that were spatially localized, oriented, and bandpass, meaning selective for a specific range of spatial frequencies. These properties closely matched the receptive fields of simple cells in the primary visual cortex. Receptive field refers to the region of the visual scene a neuron responds to and the pattern of stimulation it is tuned for. The network had no information about biology. It arrived at the same solution evolution produced in the visual system by optimizing the same objective. The nearly five decades between Shannon’s 1948 paper and the 1995 infomax algorithm was a gap in mathematical tools, not in the underlying idea.
What does the infomax algorithm actually optimize?
The cocktail party problem has a mathematical formulation that makes the challenge precise. A set of n observed signals x = (x₁, …, xₙ) are linear combinations of n unknown independent source signals s = (s₁, …, sₙ). The mixing is described by a matrix A, so that x = As. The goal is to find an unmixing matrix W such that y = Wx recovers the original sources. The recovery allows for differences in amplitude or ordering, but the signals themselves are correctly separated. It is ill-posed, meaning it has no unique solution without additional constraints. Neither A nor s is observed, so the algorithm must infer W from the statistical properties of the observations alone.
The infomax principle defines what the algorithm optimizes. The objective is to maximize the entropy of the output of a nonlinear function applied to the linearly transformed observations, where entropy measures how spread out or unpredictable the output distribution is. The nonlinearity introduces sensitivity to higher-order statistics, which is what separates sources with non-Gaussian distributions. Higher-order statistics describe structure in the data that goes beyond variance and covariance, such as whether a distribution is skewed or has heavy tails. Without the nonlinearity, maximizing output entropy reduces to a whitening operation, a linear transformation that removes correlations between signals and normalizes their variance, capturing nothing beyond those second-order relationships. Bell and Sejnowski showed that this entropy maximization objective is equivalent to maximizing the mutual information between the input and output of the network. This equivalence is where the name infomax originates. Maximizing output entropy under the infomax framework is also equivalent to finding the W that maximizes the likelihood of the observations under a model assuming the sources are independent and non-Gaussian.
The central limit theorem explains why the problem is solvable at all. The central limit theorem is the statistical principle that the sum of many independent random variables with finite variance converges to a Gaussian distribution. A mixture of independent random variables therefore tends to be more Gaussian than any of its individual components. A Gaussian distribution is the statistical signature of a mixture, and non-Gaussian distributions are the signature of sources. The unmixing matrix W is found by searching for a transformation whose outputs are as non-Gaussian as possible. When the outputs are maximally non-Gaussian, they correspond to the original independent sources as closely as the linear model allows. This is why ICA requires non-Gaussian sources and cannot separate signals that are already Gaussian, since a Gaussian mixture is statistically indistinguishable from its components.
The optimization proceeds in two steps. Bell and Sejnowski introduced the natural gradient, a modification of the standard gradient descent update that accounts for the geometry of the parameter space. Standard gradient descent treats all directions in parameter space as equally important. The natural gradient adjusts for the fact that equal steps in parameter space can correspond to very different changes in the model’s behavior. This adjustment uses the Fisher information matrix, a quantity that measures how sensitive the model’s output distribution is to changes in each parameter, effectively rescaling the update to reflect the true curvature of the optimization surface. The resulting update takes the form
ΔW ∝ [I + φ(y)yᵀ]W, where y is the current output of the network, I is the identity matrix, and φ is a nonlinear function derived from the assumed source distribution. The identity matrix anchors the update so that when the outputs are already independent, the correction term φ(y)yᵀ vanishes and the update leaves W unchanged. Before applying this update, whitening the data reduces the number of free parameters the algorithm must handle. Whitening removes second-order correlations and normalizes variance, leaving only the rotational component of the unmixing transformation for ICA to find, meaning the part that reorients the data without stretching or compressing it. This two-step procedure makes the algorithm computationally tractable and numerically stable on real data.
Why did nonlinear ICA remain unsolved for three decades?
Linear ICA is a well-posed problem, meaning it has a unique solution that the algorithm is guaranteed to find, because of two conditions. The mixing function must be linear, meaning the observed signals are weighted sums of the sources with no more complex interactions, and at most one source can have a Gaussian distribution. Under these conditions, the unmixing matrix W can be recovered uniquely. Without this guarantee, there is no principled reason to trust the components the algorithm returns.
Extending ICA to nonlinear mixing breaks this guarantee, and no algorithm can restore it without additional constraints. If the function that mixes the sources is allowed to be nonlinear, meaning the sources interact in more complex ways than weighted sums, the problem becomes unidentifiable. Unidentifiable means that for any observed data distribution, there exist infinitely many ways to write it as a nonlinear transformation of independent sources. A representation with statistically independent components under one nonlinear transformation will correspond to a representation with statistically dependent components under a different one. Without additional constraints, there is no basis for choosing between them. Unidentifiability is a structural property of the problem, not a limitation that better algorithms could overcome.
The generative modeling community arrived at the same barrier from a different direction, without initially recognizing the connection to ICA. Variational autoencoders (VAEs) became a standard framework for generative modeling. A VAE trains two networks together, an encoder that compresses an input into a small set of numbers in a latent space and a decoder that reconstructs the original input from those numbers. The latent prior, the assumed distribution over those numbers before any data is seen, is typically a standard Gaussian, and a regularization term pushes the learned latent distribution toward that prior.
A dataset of face images makes the limitation of this setup concrete. A VAE trained on faces learns to compress each image into a point in the latent space and reconstruct it accurately. In an ideal disentangled representation, one latent dimension would control hair color, another would control age, another would control whether the subject is smiling, and so on. Standard VAE training does not produce that separation. Each latent dimension ends up reflecting a mixture of these properties, and adjusting a single dimension changes multiple features of the reconstructed face at once. The model reconstructs faces accurately but the latent dimensions do not correspond to anything interpretable on their own.
Researchers who wanted disentangled representations, where each latent dimension corresponds to a single meaningful property of the data, found that standard VAE training did not reliably produce them. Beta-VAE and its successors added stronger independence penalties but without a theoretical guarantee. The nonlinear ICA problem sat in the background of all this work, but the connection was rarely made explicit.
The clean theory of linear ICA and the unresolved problem of nonlinear ICA remained separated for roughly three decades. The tools that would close the gap came from combining the VAE framework with the insight that additional observed variables could restore identifiability.
Are disentangled representations the nonlinear ICA problem?
The connection between disentangled representation learning and nonlinear ICA became formal when Khemakhem et al. showed in 2020 that a specific class of VAE could solve the nonlinear ICA problem with a guarantee. The model, called iVAE, extends the standard VAE by conditioning the prior on an observed auxiliary variable u. Conditioning here means that instead of using the same fixed prior distribution for all inputs, the prior changes depending on the value of u, such as a class label, a time index, or a domain identifier. Returning to the face dataset, if u is a label indicating whether the subject is smiling, the prior shifts depending on that value, giving the model an anchor to separate expression from other properties of the face. Under this conditioning, the latent variables become identifiable up to a component-wise transformation, meaning each recovered source may be individually rescaled or shifted, but the sources themselves are uniquely determined. This was the first result showing that nonlinear ICA could be solved with a guarantee.
Get Eleventh Hour Enthusiast’s stories in your inbox
Join Medium for free to get updates from this writer.
This result reframes a large body of prior work on disentangled representations in light of the nonlinear ICA problem. Beta-VAE and related models attempted to learn latent spaces where individual dimensions corresponded to independent generative factors, motivated by interpretability and controllability, where controllability means the ability to manipulate a single property of the output by adjusting a single latent dimension. A latent space where one dimension controls expression and another controls age is more useful than one where these factors are entangled. Without a theoretical foundation, these models could not guarantee that their learned representations were truly disentangled. The Khemakhem et al. result provides that foundation by connecting the optimization objective to the identifiability conditions of nonlinear ICA. The auxiliary variable u serves the same mathematical role as the non-Gaussianity assumption in linear ICA. It breaks the symmetry that would otherwise make the sources unrecoverable.
The practical implication for generative modeling is that identifiability turns interpretability from a desirable property into a mathematical guarantee. With the face dataset, this means a model trained with iVAE would recover latent dimensions that correspond to hair color, age, and expression as independent factors. A different model trained on the same data would recover the same dimensions up to a fixed rescaling, not a different entangled mixture. This is a stronger claim than what beta-VAE could make. The dimensions are not just penalized toward independence. They are identified in the statistical sense, meaning the same sources are found regardless of which model was used or how it was initialized.
The VAE framework also brings tools to nonlinear ICA that linear ICA never had. Deep nonlinear decoders, the part of the network that maps from the latent space back to the data, allow the model to represent complex data such as images and text. Amortized inference is the practice of training a single network to approximate the solution to an optimization problem across all inputs, rather than solving that problem separately for each one. In a VAE this network is the encoder, which learns to map any input directly to its latent representation, making training fast and scalable. The auxiliary variables that make the problem identifiable are often naturally available in practice. Time series data carries temporal structure, labeled datasets carry class information, and data collected across different environments carries domain labels. Each of these provides the conditioning variable that makes the latent factors recoverable, which is why iVAE and its successors have found applications across a range of structured generative modeling problems.
Does attention solve the same decomposition problem as ICA?
The comparison between attention and ICA becomes meaningful when the core operation in ICA is stated clearly. The algorithm passes a set of observed signals, each a linear combination of unknown independent sources, through a learned matrix that recovers those sources. The algorithm infers this matrix from the statistical structure of the data alone, by finding a transformation whose outputs are as statistically independent as possible. Knowing the value of one output carries no information about the value of another.
Attention in a transformer faces a recognizably similar problem, though it operates on a different mathematical structure. A token’s representation in a transformer layer carries mixed information from surrounding context, from positional relationships, and from features the network has learned to encode in earlier layers. The attention mechanism decomposes this mixed representation by computing, for each query vector, a weighted combination of value vectors. The weights come from comparing the query against a set of key vectors using a scaled dot product followed by a softmax, where softmax converts a set of raw scores into a probability distribution so that the weights sum to one. High similarity between a query and a key produces a large weight, pulling the corresponding value into the output. At each position, the output blends contributions from across the sequence in proportion to their relevance, decomposing the token’s mixed representation into a selective combination of information drawn from different positions.
The two methods share the same underlying problem structure while differing in how they solve it. ICA relies on higher-order statistics and maximum non-Gaussianity. Attention relies on learned dot-product similarity and gradient descent, the standard procedure for updating network parameters by following the direction that reduces the training loss. ICA finds a fixed unmixing matrix from a batch of data. Attention computes a different decomposition for every input, with queries and keys adapting continuously during training. Both methods receive a representation that mixes multiple sources of information and produce an output that separates or weights those sources according to a learned criterion. ICA identifies contributions by their statistical independence. Attention identifies them by learned relevance.
Multi-head attention makes the structural parallel to ICA more explicit. A single attention head computes one decomposition of the mixed input representation. Multiple heads, applied in parallel to the same input with different learned projections, compute multiple decompositions simultaneously. Each head attends to different aspects of the input, specializing in different types of relationships. The outputs of all heads are concatenated and projected back into the model’s representation space. This is structurally close to ICA finding multiple independent components from the same mixture. Each component captures a different mode of variation in the data, meaning a distinct pattern along which the data changes. The difference is that ICA imposes statistical independence as a hard constraint derived from the data distribution, while attention heads differentiate through gradient descent on a task-specific loss.
Empirical work on trained transformers adds a concrete dimension to this structural parallel. Analysis of attention head behavior in large language models has found that individual heads specialize in ways that are not arbitrary. Certain heads consistently track syntactic dependencies, meaning they attend to words that are grammatically related regardless of their distance in the sentence. Others follow coreference chains, linking pronouns to the nouns they refer to across long distances. Others attend to positional patterns, favoring adjacent tokens or fixed offsets. This specialization emerged from training on language modeling objectives without any explicit instruction to decompose representations into functionally distinct channels. Trained attention heads empirically arrive at something resembling the independent components ICA extracts from a mixed audio signal, through a different mechanism and toward a different objective, while solving a recognizably similar problem.
What happened to the cocktail party problem in the deep learning era?
The cocktail party problem survived the deep learning era intact. The task of recovering individual voices from mixed recordings without knowledge of the mixing process remained the same. What changed was how it was solved. ICA-derived methods built around frequency-domain analysis became standard in recording studios, broadcasting, and hearing aids, where separating a target voice from background noise is a practical requirement rather than a research exercise.
Conv-TasNet, introduced by Luo and Mesgarani in 2019, approached the problem differently from ICA. The original ICA method operated in the frequency domain, meaning it first converted the raw audio signal into a spectrogram, a representation that shows which frequencies are present at each moment in time, and then separated the sources in that representation. Conv-TasNet operates directly on the raw waveform, the audio signal as a sequence of numbers over time, without any conversion step. The network passes this signal through a learned encoder that produces a compact feature representation. A stack of dilated convolutional layers, meaning convolutional layers where the filter skips over inputs at regular intervals to capture patterns across longer time spans, then estimates a mask for each source. A mask is a set of weights that selectively retains the parts of the representation belonging to one speaker while reducing the contribution of the others. The masked representations are decoded back into waveforms, and the entire pipeline is trained end to end with a loss that measures how closely the separated signals match the true clean sources.
The performance gap that emerged revealed something important about where the structure of the problem actually lives. Conv-TasNet surpassed the ideal time-frequency magnitude mask on standard speech separation benchmarks. The ideal mask is an oracle, a solution computed with full knowledge of the true clean signals that cannot be used in practice since those signals are what the algorithm is trying to recover. Surpassing it means the time-domain approach found structure in the raw waveform that frequency-domain analysis, even with access to ground truth, could not exploit.
This shift from frequency domain to time domain connects back to a core assumption in the original ICA framework. ICA assumed that independence structure lived in the frequency domain, because different speakers tend to occupy different frequency ranges at different moments and the spectrogram makes this structure visible. Conv-TasNet learns its own representation of the input from scratch, without assuming where the independence structure lives. The learned encoder finds whatever structure in the raw waveform is most useful for separation. This turns out to be richer than what a fixed time-frequency transform, a predetermined mathematical operation that converts a signal from the time domain into a frequency representation, captures. What ICA accomplished by exploiting known statistical properties, Conv-TasNet accomplishes by learning those properties from data.
Do modern networks rediscover independent components on their own?
A different kind of connection between ICA and deep learning exists at the level of what trained networks actually learn, and it emerges from applying the tools of ICA to internal representations of networks that were never asked to do anything related to ICA. When ICA is applied to the activations of convolutional networks trained on natural images, the result reveals interpretable structure that was not explicitly requested. Activations are the output values of neurons at each layer of the network, the internal numbers the network computes as it processes an input. The independent components recovered from early layer activations are spatially localized, oriented, and bandpass, meaning they respond to edges at specific angles and specific scales, closely matching the Gabor-like filters that ICA extracts directly from natural image patches. Gabor filters are mathematical patterns that look like oriented stripes within a smooth envelope, and they closely resemble the receptive fields of simple cells in the primary visual cortex. The network was trained to classify images using cross-entropy loss, a standard measure of how far the network’s predictions are from the correct labels, optimized through backpropagation, the standard algorithm for computing how much each parameter contributed to the error so that parameters can be updated accordingly. The independent structure appeared without being requested.
This finding connects the classification objective back to Barlow’s 1961 efficient coding conjecture and to the 1995 infomax result. If efficient coding of natural image statistics leads to ICA-like filters, and if gradient descent on image classification also leads to ICA-like filters, then both objectives arrive at similar internal representations, one driven by statistical structure and one driven by task performance. The statistical structure of natural images appears to guide networks toward representations that are independently coded in an approximate sense, regardless of what objective is being optimized.
Analysis of deeper network layers using ICA and related decomposition methods has found that this pattern extends beyond early filters. A component here means a direction in the space of activations, where activations form a high-dimensional space with one axis per neuron, that captures a distinct mode of variation in the data, similar to how an independent component in ICA captures a distinct source signal. Some components correspond to texture detectors, others to object parts, others to responses that align with semantic categories. The components at each depth are not perfectly independent, but they are more independent than the raw activations and more interpretable than individual neurons taken in isolation. This pattern suggests that training pushes representations toward a basis, meaning a set of directions that together span the space of possible activations, that is approximately independent without any explicit pressure to do so.
Conclusion
The deeper contribution of Bell and Sejnowski’s 1995 infomax paper was to show that efficient neural coding, signal separation, and information maximization were descriptions of the same problem in different languages. Nearly five decades of signal processing research and a conjecture from visual neuroscience converged in a single optimization objective trained on a neural network. That convergence turned out to be more than a historical coincidence.
The problems ICA identified as central have reappeared in new forms across modern machine learning. Attention mechanisms perform learned decomposition of mixed representations across sequences. Variational autoencoders, when conditioned on auxiliary variables, solve the nonlinear version of the same blind separation problem with a guarantee. Conv-TasNet and its descendants apply the tools of deep learning to the same cocktail party problem Bell and Sejnowski were working on, outperforming what any frequency-domain method can achieve. Networks trained on classification tasks develop approximately independent internal representations without being told to.
The algorithm also found its way back into the field that partly inspired it. Sejnowski’s laboratory used ICA to remove movement artifacts from mobile EEG recordings, where EEG, or electroencephalography, measures electrical activity produced by the brain through sensors placed on the scalp. Allowing the subject to move freely during recording introduces noise from muscle activity and other non-neural sources that overlap with brain signals. ICA separates these mixed signals, allowing subjects to walk freely in a virtual environment while their brain activity is measured accurately. The same unmixing operation that separated voices in a room separated neural signals from muscle noise in a brain, both a problem the nervous system poses and one it helps solve.
References:
Bell, A. J., & Sejnowski, T. J. (1995). An information-maximization approach to blind separation and blind deconvolution. Neural Computation, 7(6), 1129–1159. https://doi.org/10.1162/neco.1995.7.6.1129
Khemakhem, I., Kingma, D., Monti, R., & Hyvarinen, A. (2020). Variational autoencoders and nonlinear ICA: A unifying framework. Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics. https://arxiv.org/abs/1907.04809
Luo, Y., & Mesgarani, N. (2019). Conv-TasNet: Surpassing ideal time-frequency magnitude masking for speech separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(8), 1256–1266. https://doi.org/10.1109/TASLP.2019.291516
Sejnowski, T. (2018). The Deep Learning Revolution. MIT Press. https://mitpress.mit.edu/9780262038034/the-deep-learning-revolution/
